Credentials were Reset

[transcendr]

Describe the bug
Been using flowise for a few months now with no major issues. Today the system seemingly reset all saved credentials. I didn't notice any other data loss and readding the credentials, updating the flows worked as expected.

To Reproduce
(unavailabe)

Expected behavior
Credentials are persisted indefinitely

Screenshots
(not applicable)

Flow
(system, not flow level)

[Luizrebelatto]

I had the same problem

[mickeykootwct]

Same issue on this side happend a few times today.

[transcendr]

This has happened twice since my posting this and at more frequent intervals. Can any contributor suggest a workaround while waiting for a solution. Readding the keys every few hours is untenable (lol). Also I'm unclear as to why this is suddenly an issue. I have not updated flowise for fear of breaking what's been working just fine for us, so this isn't a bug that's been introduced lately.

@Luizrebelatto had as in past tense? Is it still persisting. Did you find a valid workaround?

[mickeykootwct]

@transcendr kind of depends on the situation but you can also push the credentials via the API if you use any external code to run it. That way you don't need the credentials in flowwise.

[transcendr]

@mickeykootwct thanks, I'm using two basic conversation chain chat models, one for open ai and one for google vertex, where you select the credentials from the dropdown. Are you suggesting something like using the custom JS function node to call the respective API's directly instead of the langchain wrappers, or something else?

[Luizrebelatto]

This has happened twice since my posting this and at more frequent intervals. Can any contributor suggest a workaround while waiting for a solution. Readding the keys every few hours is untenable (lol). Also I'm unclear as to why this is suddenly an issue. I have not updated flowise for fear of breaking what's been working just fine for us, so this isn't a bug that's been introduced lately.

@Luizrebelatto had as in past tense? Is it still persisting. Did you find a valid workaround?

i had this problem last week and yesterday
I manually created the credentials again, but I can't keep creating them all the time, otherwise it's a very bad process to maintain my chatbot. I haven't found a solution yet

[HenryHengZJ]

Hey guys, we've investigated this and concluded this is an attack by bad players. But rest assured, no credentials were stolen because in order for hackers to get the real api keys, they would need an encryption key to decrypt it. And that encryption key is not stored on database, but on your deployed instance file system.

This attack only started after we merged this PR (#1812) which has a swagger file. Most likely some malicious script is written to scrape the urls and try to hit the credentials apis. This can be proven by the logs:

If you notice, the time for every request is almost identical.

Since then we have updated the swagger spec and implemented tighter url checks to prevent this from happening. Please update to the latest version 2.0.6.

Please report back if issues still persists

[transcendr]

@HenryHengZJ thanks so much! Our's is deployed on GCP kubernetes. To redeploy the updated version without interruption, is it as simple as rebuilding 2.0.6 and pushing it to the GCP container registry? Will the encryption key stored on the filesystem persist in this case?

[HenryHengZJ]

Yes simply pull from latest docker image. Encryption key will persists if you have your SECRETKEY_PATH set: https://docs.flowiseai.com/configuration/environment-variables#for-credentials

[transcendr]

@HenryHengZJ great, unfortunately this variable wasn't set in our config. I imagine this means the existing key on the filesystem now won't be persisted if the instance is redeployed. What would be the effect of that key being changed now? Is it only the credentials that depend on that key for decryption or are there other parts of the system that are encrypted with that key? Is it just a matter of recreating the creds with the new key?

[HenryHengZJ]

its only credentials that depend on that key for decryption. yes you just need to create the creds after setting the env variable to persists the storage the encryption key.

[transcendr]

@HenryHengZJ - Thanks again for your quick fix on this. As a DB I use, supabase which with the following env config was working fine, but now after updating to 2.0.6, flowise no longer connects to supabase (actual values replaced with 0's):

In the logs, I see a bunch of errors that repeat for different components like this:

Any idea?

[HenryHengZJ]

@HenryHengZJ - Thanks again for your quick fix on this. As a DB I use, supabase which with the following env config was working fine, but now after updating to 2.0.6, flowise no longer connects to supabase (actual values replaced with 0's):

In the logs, I see a bunch of errors that repeat for different components like this: Any idea?

I think thats the different issue. Feel free to open another issue for that. Closing this for now

[daverad]

For those effected here, I wanted to share our recent experience as it may impact you as well.

Somehow our Google Vertex Service account credentials were recently compromised resulting in a significant bill (thousands $) within a few days.

While we cannot be 100% certain that the issue originated from Flowise, we have no reason to believe it was caused by anything else, as those credentials were exclusively used with Flowise. Despite the credentials being encrypted, they were somehow compromised.

Also of note, due to Issue 3177 - Cannot connect to Postgres database when deployed, we were still using an older branch of Flowise from July 2022.

Google and Anthropic (the service the bad actor used with our credentials) have both said they are unable to refund us. As a small startup up - its been a big hit and I wouldn’t want this to happen to anyone else.

So as a precaution, I recommend checking and possibly resetting your credentials, ensuring you are on the latest version of Flowise, and also setting clear budgets, limits and alerts with Google Cloud just in case.

FYI @HenryHengZJ

[HenryHengZJ]

Thanks @daverad for the sharing. Last thing I want is to have scenario like this 😔