feat(firestore-bigquery-export): Add Gemini agent to gen-schema-view

[cabljac]

This PR adds the option to use a Gemini Agent to generate the schema files, fixes some tests, and corrects a parameter description.

[Gustolandia]

Testing

Deeply Nested Documents:

No explicit test cases for collections with deeply nested fields, such as maps within arrays or arrays within maps.

Large Collections

There are no tests for large collections that exceed the agent-sample-size. A test case should simulate the agent gracefully sampling the first N documents and stopping without errors.

test("should handle collections with more than 100 documents", async () => {
  const largeSampleData = Array.from({ length: 200 }, (_, i) => ({ id: i, value: `doc${i}` }));
  const response = await runAgent(apiKey, "./schemas", "large_collection", "test_table", largeSampleData);
  expect(response).toBeDefined();
  expect(response.success).toBeTruthy();
});

[cabljac]

• Deeply Nested Documents:

  • No explicit test cases for collections with deeply nested fields, such as maps within arrays or arrays within maps.
  • Suggestion: Add a test in e2e.test.ts with a deeply nested document like:

    const sampleData = [
        {
            user: {
                profile: {
                    name: "Alice",
                    age: 25,
                },
                settings: {
                    notifications: {
                        email: true,
                        push: false,
                    },
                },
            },
        },
    ];

• Large Collections:

  • There are no tests for large collections that exceed the agent-sample-size. A test case should simulate the agent gracefully sampling the first N documents and stopping without errors.

    test("should handle collections with more than 100 documents", async () => {
        const largeSampleData = Array.from({ length: 200 }, (_, i) => ({ id: i, value: `doc${i}` }));
        const response = await runAgent(apiKey, "./schemas", "large_collection", "test_table", largeSampleData);
        expect(response).toBeDefined();
        expect(response.success).toBeTruthy();
    });

I think i'll leave these out, they're fair points but not relevant to this PR in particular.

Actually the "Large Collections" comment isn't correct; we're taking just a sample

[Gustolandia]

Actually the "Large Collections" comment isn't correct; we're taking just a sample

The current tests seem to be missing a verification for whether the sampling behavior correctly handles large collections. The tests should explicitly check that only the specified number of samples (agent-sample-size) are being processed, regardless of the total size of the collection.

What I wrote should be like this instead (apologies):

  test("should handle collections with more than 100 documents by sampling", async () => {
      const largeSampleData = Array.from({ length: 200 }, (_, i) => ({ id: i, value: `doc${i}` }));
      const agentSampleSize = 50; // Example sample size
      const response = await runAgent(apiKey, "./schemas", "large_collection", "test_table", largeSampleData);

      // Verify correct sampling behavior
      const sampledData = largeSampleData.slice(0, agentSampleSize); // Expected sample
      expect(response).toBeDefined();
      expect(response.success).toBeTruthy();

      // Assert sampled data is passed for schema generation
      expect(response.schemaGeneratedFrom).toEqual(sampledData); // Assuming `response.schemaGeneratedFrom` reflects sampled data
  });

[pr-Mais]

I think its better to include a copy-paste minimal usage here, e.g. add --use-gemini-agent and other required params

[cabljac]

We have a non-interactive example below, can you explain what you mean? confused

[pr-Mais]

should we check if the schema file already exist and exit?

[cabljac]

yes, good catch