[Feature]: Add content moderation layer using Gemini AI before anonymous posts are published to the public feed

[divyanshim27]

Summary

SafeVoice is an anonymous platform designed to provide a safe space for users to share sensitive experiences related to harassment, abuse, and discrimination. However, the absence of a content moderation layer means that the public feed is entirely dependent on user self-regulation. Without automated pre-screening, harmful content (hate speech, targeted harassment, graphic descriptions) can appear in the feed and re-traumatize the very users the platform is designed to protect.

Problem

• Posts go directly from ShareStory.tsx to Firestore without any content screening step.
• The platform already uses Google Gemini AI (via correct-grammar.cjs and translate.cjs Netlify Functions) — adding a moderation function is a natural and consistent extension of this existing architecture.
• Without moderation, bad-faith users can abuse the anonymous posting feature to post content that violates the platform's safety mission.
• There is no reporting or flagging mechanism for existing posts, so harmful content can persist indefinitely.

Impact

• Users seeking support encounter harmful or triggering content in the feed, directly harming the platform's core mission.
• The platform becomes a liability risk without any content policy enforcement mechanism.
• Trust from NGO partners (who provide resources via the NGO Resource Hub) may erode if the platform gains a reputation for unmoderated content.

Proposed Solution

I would like to add a moderate-content.cjs Netlify function that screens posts before they reach Firestore:

// netlify/functions/moderate-content.cjs
const { GoogleGenerativeAI } = require('@google/generative-ai');

exports.handler = async (event) => {
  const { content } = JSON.parse(event.body);
  const genAI = new GoogleGenerativeAI(process.env.GEMINI_API_KEY);
  const model = genAI.getGenerativeModel({ model: 'gemini-1.5-flash' });

  const prompt = `You are a content safety classifier for a support platform for survivors of harassment and abuse.

Analyze the following content and respond with ONLY a JSON object:
{
  "isSafe": boolean,
  "reason": "brief reason if unsafe",
  "category": "safe" | "hate_speech" | "targeted_harassment" | "graphic_violence" | "spam"
}

Content: """${content}"""`;

  const result = await model.generateContent(prompt);
  const response = JSON.parse(result.response.text());

  return {
    statusCode: 200,
    body: JSON.stringify(response)
  };
};

Update ShareStory.tsx to call moderation before Firestore write:

const moderationResult = await fetch('/.netlify/functions/moderate-content', {
  method: 'POST',
  body: JSON.stringify({ content: storyText })
});
const { isSafe, reason } = await moderationResult.json();

if (!isSafe) {
  setError(`This content could not be published: ${reason}. Please review our community guidelines.`);
  return;
}
// Proceed to Firestore write only if safe

Additionally, I will add a "Report Post" button on the feed that sets a reported: true flag in Firestore for admin review.

Additional Notes

• Moderation is advisory for edge cases — the Gemini prompt is tuned to be permissive for genuine survivor experiences while blocking clear-cut harmful content.
• The Firestore Admin collection can be used to store flagged posts for human review via the existing AdminDashboard.tsx.
• Rate limiting on the moderation function will reuse the existing express-rate-limit pattern.

I would be happy to implement this end-to-end. Could you assign this issue to me?

Labels: enhancement, security, feature, GSSoC 2026

[divyanshim27]

Hey @Piyushydv08 ,
Please assign it to me under GSSoC 2026.
/assign /claim /claim-assignment