This is a cross post from a research group blog post I made which can be found here

OpenObserve vulnerability chain Link to heading

TL;DR - OpenObserve deployments using version 0.9.1 or lower are vulnerable to the following privilege escalation chain:

• A malicious user submits logs via a service which sends logs to an OpenObserve instance. These logs contain malicious content.
• A site user attempts to create a dashboard, using the logging field containing malicious input.
• The malicious JavaScript is executed, exfiltrating the user’s username and password.

Technical details Link to heading

What is it? Link to heading

OpenObserve is a simple yet sophisticated log search, infrastructure monitoring, and APM solution. It is a full-fledged observability platform that can reduce your storage costs by ~140x compared to other solutions and requires much lower resource utilization resulting in much lower cost.

As one of group’s members run this software and we had some free time, we decided to have a look into the security of the solution.

What was wrong? Link to heading

Usernames and passwords in local storage Link to heading

Initially exploring the solution, we noticed that the platform uses Basic Auth as the underlying authentication scheme. Basic authentication is essentially when a website includes a user’s username and password within all requests in order to authenticate with a backend server. Due to this, the website needs to store these credentials somewhere accessible to the front end, and thus accessible to essentially anyone.

Further to this, the credentials are stored in local storage! As an attacker this is great because client side JavaScript is freely allowed to retrieve these credentials at any time. As such, if we can get cross site scripting (XSS), we have access to the account creds :)

These credentials can be seen in the below screenshot of a local deployment:

We want your account creds please Link to heading

In order to find XSS and leak credentials, a simple place to start is often with static analysis tools. Static analysis tools essentially offer a user the ability to audit a code base for vulnerabilities without the need to run the software, or have a pentester background. These tools generally just run over the files, often running pre-made plugins to look for common vulnerabilities. While generalised, these tools often finding amazing bugs and allow both attackers and defenders the ability spend more time ‘doing the thing’ and less time ‘finding the thing’.

As OpenObserve is written in a combination of Rust and JavaScript, we decided to start with a tool called Semgrep which provides the ability to audit essentially any language.

Using Semgrep (semgrep scan --config auto) we noticed the following issue popping up in a few places:

❯❱ javascript.vue.security.audit.xss.templates.avoid-v-html.avoid-v-html
      Dynamically rendering arbitrary HTML on your website can be very dangerous because it can easily lead to XSS vulnerabilities. Only use HTML interpolation on trusted content and never on user-provided content.
      
      Details: https://sg.run/0QE

Essentially this issue denotes that Semgrep has noticed that the client side code used by OpenObserve may be using user provided data unsafely. As outlined in the code block, this client side code uses a variable which could possibly be set by an end user, and if so then the likelihood of XSS is high.

Now, given the amount of theoretically untrusted data in the platform (every log!), this seemed like a great place to start investigating the possibility of XSS.

Tracing the Semgrep alerts back, we eventually discovered that OpenObserve supports the ability to create dashboards using conditional formatting based off of log content. We’d guess this is for things like filtering by severity, but in our case it’s a good avenue for XSS due to untrusted input being used within v-html. An example implementation looks something like the following where opt is user supplied input:

<div v-html="opt"></div>

In practice, this looks something like the following:

Where each of the items in the drop-down list are set using the prior code block.

So we now know the following about the site:

• The website inserts untrusted input directly into the HTML
• We have found where in the UI to trigger this

So in order to exploit these issues, we need to create a proof of concept (PoC). This PoC should ideally showcase the execution of malicious client side code while also demonstrating a security impact. In the end we came up with the following Python script:

import json

import httpx

payload = {
    "level": "info",
    "ingestion": "<img src=null onerror=\"javascript:alert(localStorage.getItem(\'access_token\'))\"></img>"
}

auth = httpx.BasicAuth(username="REDACTED", password="REDACTED")
client = httpx.Client(auth=auth)
resp = client.post(
    "http://DOMAIN_HERE/api/default/default/_json",
    content=f'[{json.dumps(payload)}]'
)

print(resp.text)

This PoC sends a malicious piece of JavaScript that when executed, grabs the current account username and password before printing it to the screen. Further iterations of this script would be able to send this data to an attacker, etc.

The execution of this payload can be seen in the screenshot below:

And for those of you at home, that alert translates to:

Which are the credentials we used to start this docker image:

A more realistic, alternative path to account takeover Link to heading

Now let’s look at a more realistic, alternative injection route. We want you to imagine that you run various applications and websites. As you follow general best practices, you use a centralized logging service in order to aggregate logs and ensure they are available for review. As a part of this, all requests to your web applications are logged in a format similar to below:

{
  "request_method": "GET",
  "request_path": "/admin",
  "request_protocol": "http",
  "request_domain": "localhost",
  "request_query_parameters": "",
  "request_origin_ip": "192.168.0.1",
  "request_user_agent": "User agent goes here"
}

Now that log lets you know that a GET request was made to http://localhost/admin by a user originating from 192.168.0.1. This is great, although we are slightly concerned by the request being made to /admin. Why don’t we create a dashboard panel tracking how many requests are made to /admin?

So following the prior steps, you create a panel and select the request_path field to filter by. Now we only want to count the requests to /admin so we need to open the filter and select this item. But what!? A message has just popped up on my screen displaying my credentials in Base64.

The reason for this is relatively simple, someone made a GET request to your website at the following path and due to the vulnerabilities described here your credentials have now been leaked. In this example, they only popped up on your screen but in a malicious example they may have silently be exfiltrated to an attacker.

The URL the GET request was made to that results in the credentials being displayed: http://localhost/admin<img src=null onerror="javascript:alert(localStorage.getItem('access_token'))"></img>

Now that is quite an exploit chain, and it’s one which is made worse due to the unique position in which a centralized logging service sits. It’s also definitely not every day we see cleartext passwords, and even less often that we can access them via client side injection. Obviously there’s more at fault here then a single issue.

Good practice Link to heading

There are a couple key issues here which allowed this exploitation chain. Preventing any one of them would have made this exploit significantly harder if not impossible.

1. Log’s received from ingestion sources should not be trusted. OpenObserve partially used DOMPurify, but there were a few areas where it was not implemented.
2. OpenObserve did not use sessions. Instead, the username and password was stored in local storage and was used through Basic Auth. We recommend removing the need for basic auth and implementing better session management practices. These best practices can be found here.

Disclosure timeline Link to heading

• 16/03/2024: Disclosure submitted to OpenObserve via GitHub security disclosure pipeline.
• 28/03/2024: Email sent requesting acknowledgment that the issue has been seen.
• 29/03/2024: Acknowledgment received.
• 21/04/2024: Status update requested.
• 22/04/2024: OpenObserve acknowledges that the fixes have been pushed to the repository and mentions the advisory will be published within a week.
• 22-23/04/2024: Various discussions surrounding blog posts.
• 8/05/2024: OpenObserve provides information on technical mitigations introduced to resolve the underlying issues.
• 14/06/2024: Publication status update requested.
• 19/06/2024: OpenObserve mentions the advisory will be published the following week.
• 7/07/2024: Publication status update requested.
• 13/07/2024: OpenObserve mentions that a release has not occurred in quite a while, but it will go live as soon as a new release is completed.
• 13/07/2024: 90 day disclosure notice given.
• 26/07/2024: Advisory published and CVE assigned.
• 5/08/2024: Blog post published.