Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Data type errors for converted rules #103

Open
abulhol opened this issue Dec 3, 2024 · 2 comments
Open

Data type errors for converted rules #103

abulhol opened this issue Dec 3, 2024 · 2 comments

Comments

@abulhol
Copy link

abulhol commented Dec 3, 2024

Using:
Elasticsearch 8.16.1
pysigma-pipeline-sysmon==1.0.4
Winlogbeat 8.16.1 on Windows 10

I converted all the rules from sigma/rules/windows in the SigmaHQ repository to EQL like so:

    pipeline = sigma.pipelines.elasticsearch.ecs_windows()
    backend = es.EqlBackend(pipeline)

   for root, dirs, files in os.walk(args.indir):
        for filename in files:
            if filename.endswith(".yml"):                
                with open(os.path.join(root, filename), 'r') as f:
                    try: 
                        rule = sigma.rule.SigmaRule.from_yaml(f)
                        eql_query = backend.convert_rule(rule)
                        with open(os.path.join(args.outdir, filename.rsplit('.', 1)[0]), 'w') as o:
                            o.write(json.dumps({
                            "query" : eql_query[0]
                            })) 

Now when I run the 2277 resulting rules against my Elasticsearch index (coming from Winlogbeat on Windows 10), a total of 787 rules fail.
A frequent error I get for these is:
[:] cannot operate on first argument field of data type [text]: No keyword/multi-field defined exact matches for [script_block_text]; define one or use MATCH/QUERY instead
For example for:

  • posh_ps_xml_iex.yml
  • posh_ps_invoke_obfuscation_via_use_rundll32.yml

Am I doing something wrong in the conversion code?

@abulhol
Copy link
Author

abulhol commented Dec 4, 2024

For references, this is the query that the converter produces and which I try to run against the ES index:

{
  "query": "any where (powershell.file.script_block_text:\"*New-Object*\" and powershell.file.script_block_text:\"*System.Xml.XmlDocument*\" and powershell.file.script_block_text:\"*.Load*\") and (powershell.file.script_block_text like~ (\"*IEX *\", \"*Invoke-Expression *\", \"*Invoke-Command *\", \"*ICM -*\"))"
}

@abulhol
Copy link
Author

abulhol commented Dec 4, 2024

I investigated a bit further:
The issue above seems to be that powershell.file.script_block_text is of type text but has no keyword field (also according to the ES mapping).
The general issue seems to be that field types between the index generated by Winlogbeat do not work with the EQL queries. Here is another example:
second argument of [:] must be [string], found value [4697] type [integer]

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant