Blog

Release announcements, helpful tips, and community discussion

9.5

•

Cerb (9.5) is a security update released on March 03, 2020. It includes more than 37 new features and improvements from community feedback.

Cerb 9.5 is a free update for *all* active and expired self-hosted licenses at your current seat count. We do not feel it would be ethical to hold security updates hostage in order to promote renewals. Contact us for an updated license.

Important Release Notes

Security Hardening

This update includes several security enhancements uncovered during proactive internal code audits. We have not received any related reports from third parties, nor have we seen any evidence of exploitation. Nevertheless, it is still possible these issues were previously known outside of our team.

A brief primer in web security

Cerb has more than 500 “endpoints” where a client browser can send a request for the server to provide data or perform an action. For instance: filter a worklist, add a decision branch to a bot behavior, start replying to a ticket, save a draft, create a task, edit a worker, log in, ping a webhook, and so on. Many of these endpoints are similar, like creating and editing various record types.

While this may seem like a lot of virtual doors and windows to protect, Cerb is designed following a common architectural design pattern called “Model-View-Controller” (MVC). The primary intent of MVC is to separate data (model), presentation (view), and logic (controller). Put simply, this means when you navigate within Cerb from your web browser, there isn’t a real folder or file on the server for these locations.

Instead, nearly every path in Cerb is “virtual” – requests enter in only two possible ways, and all requests start through the same route. For instance, the first step of this route could require an authenticated user or redirect to the login form. That protection would happen in one place for all current and future functionality, rather than in 500+ places, where it would be conceivable for a developer to forget at least once, and a tester/reviewer to miss at least once.

Endpoints are organized into “controllers”, which group related endpoints together and control access. Most controllers require an authenticated worker to use any of their endpoints. Some controllers, like “config”, require an administrator account to use. Most endpoints further require specific permissions (e.g. editing a ticket record). The main exceptions are login, resources, webhooks, and portals, which interface with unauthenticated users.

Cerb has a comprehensive role-based permissions system. However, protecting these endpoints from unauthenticated or unprivileged access is only the first (and perhaps simplest) step in web security.

Common attacks against web applications attempt to trick an already authenticated user into performing actions within their own browser. This doesn’t require an attacker to know secret details like a user’s password. From the server, these requests may be indistinguishable from legitimate requests.

Two of the most common forms of these attacks are:

  • Cross-site request forgery (CSRF): An attacker redirects your browser from a third-party site (a malicious link) to submit a forged request against an application where you are already authenticated. It is important to keep in mind that the attacker doesn’t see the result of these requests. They can theoretically trick your browser into performing a sequence of actions that give them more direct access (depending on your permissions). This is one major reason why most applications require you to enter your current password to set a new password (or verify a one-time code, or CAPTCHA). Someone can’t trick your browser into changing your password to something known by the attacker if they don’t already know your current password.

  • Cross-site scripting (XSS): Websites are built with interpreted languages like HTML and Javascript. With XSS vulnerabilities, an attacker sends malicious content that is unintentionally executed along with normal content when viewed by your browser. Unlike CSRF attacks, XSS attacks can easily “exfiltrate” (copy/send) sensitive information back to the attacker. An XSS attack can subvert CSRF protections.

Cerb uses several strategies to mitigate XSS attacks. The primary one is “escaping”, where all user-provided content displayed in the browser is automatically shown literally rather than interpreted and executed (as HTML or Javascript). Similarly, there is “sanitization”, which removes potentially malicious features from user-provided content (all scripting, some formatting, and some methods of fetching data). The latter is how we display HTML email, where preserving formatting is desirable, but running arbitrary scripts is not.

Cerb also uses several strategies to mitigate CSRF attacks. The primary one is a secret “token” that must be provided with all data-manipulating requests. An attacker could trick your browser into performing an action as you, but they wouldn’t know the secret token and Cerb would reject the request.

Browsers also enforce basic “same origin” protection, simply meaning a subsequent request should come from the same website you’re currently looking at. These defenses are trivial to overcome by using developer tools to forge requests (which would lack authentication), but they make it more difficult to trick authenticated users into performing unexpected actions from within their own browser.

Effective web security requires several layers of protection. For instance, if some CSRF protection can be avoided through flaws or new attacks, the application should offer other defenses to reduce the impact of those potential vulnerabilities.

For a practical example, let’s consider the two most common request types (methods/verbs) that a browser makes:

  • A GET requests read-only information (e.g. viewing a message, comment, or article).

  • A POST submits data that makes changes (e.g. replying, adding a comment, closing a ticket).

In a well-designed application, a GET request never performs actions that modify data. In this scenario, CSRF attacks against GET requests have no effect because the attacker can’t see the response and the server performs no actions (consequently they often don’t require secret tokens). POST requests require the secret token, and a CSRF attacker shouldn’t be able to discover it (without a corresponding XSS vulnerability).

These measures work together to improve security.

Disclosure

During our code audit, we discovered an unconventional way to circumvent CSRF protection for some GET requests. This could allow an attacker, who had enough information, to trick a worker’s browser into sending forged requests, which could potentially change data. An attacker wouldn’t be able to directly see any results from their forged requests, and in most cases they would need to know internal IDs to make coherent changes or cover their tracks.

In the same day we discovered the issue, we patched Cerb Cloud, we deployed a stopgap patch to GitHub, and we notified all recent self-hosted license owners.

During our continued testing, it wasn’t possible to upload files, create or modify worker accounts, or import arbitrary packages. This prevented potentially serious compromises. It was, however, possible to do things like blindly creating comments on random records as the current worker, which could theoretically have been used for phishing (deceptive links from seemingly trusted sources) or social engineering attacks (pose as a known manager/executive asking for something to be done).

In practice, this would likely require a high degree of familiarity with a target environment for a convincing deception. It would also leave a trace, and workers would probably report not having authored comments. The comments would be on random records due to the attacker having no idea which IDs were valid or what their content was. Perhaps, theoretically, some of this information could be gleaned from legitimately opened tickets in the Support Center portal, or in cooperation with a compromised/malicious worker. As best we can tell, it would be very difficult to do from the outside.

We scanned the past several months of Cerb Cloud request logs and couldn’t find any evidence of forged requests. Our testing always left obvious markers.

The flaw is rooted in the fact that many of our endpoints have been using a relaxed method of reading request parameters, allowing either GET or POST.

This was expected behavior, because our CSRF protection was also being enforced against most GET requests (requiring a token), in order to make it easier for users to use our provided scripting to create dynamic widgets, bot interactions, and automations. For instance, a widget that provided a shortcut button could fetch a simple URL and get back either some data on success, or an error.

This complacency had spread well beyond what was required by widgets or bot interactions, with the assumption that the CSRF protection was adequate. In other words, we were attempting to be more clever than a core tenet of web security.

Resolution

First, we corrected the underlying flaw in the CSRF protection.

Then we concluded that the convenience of widgets and bot interactions being able to automate some actions with GET requests wasn’t worth the potential for future abuse.

In the modern web, dynamically constructing POST requests from scripts has become much simpler. We no longer support older browsers that lack these features (e.g. Internet Explorer). This wasn’t true a few years ago.

We manually audited all 550+ endpoints in Cerb to ensure that any request resulting in data modification always required a POST request, CSRF token, and that no parameters were accepted from the URL.

We rewrote our request routing code to make it much harder for developers to accidentally create endpoints that work in the wrong context.

While we were auditing all endpoints, we also took the opportunity to proactively verify that every endpoint was using our latest permissions functionality, and also couldn’t be abused by authenticated workers.

Out of 550+ endpoints:

  • We found 3 endpoints where authenticated workers could craft manual requests (outside of the UI) to view information they lacked access to.

  • We found 4 endpoints where authenticated workers could craft manual requests (outside of the UI) to perform minor actions they lacked permission for. These were difficult to abuse, but still should not have been possible.

  • We found 2 instances where we could better harden the security of session cookies.

These issues were all resolved.

Due to the significance of these improvements, and the difficulty of creating similar patches targeting older versions, we have decided to make Cerb 9.5 a free update for all current and expired self-hosted licenses.

We are also including some bonus privacy and security related features that we had planned for the next update.

Email Privacy and Security

The display of HTML email has been overhauled for privacy and enhanced security.

Block external images from untrusted senders

When displaying an HTML message, a report is now displayed above the message with the total number of images and links, along with the number of each that are blocked. Clicking these totals displays a popup with the list of external links for easy review. This also naturally draws attention to privacy abusers. In our testing, we’ve seen messages from seemingly trustworthy brands with over 20 tracking images (likely shared with advertisers).

Images are now automatically disabled when sent by new or untrusted senders. This makes privacy the default. [#1194]

When displaying an HTML message, images can be displayed once, or always displayed for a given trusted sender. A sender can quickly be flagged as trusted from the images summary popup. Trust may also be revoked in the same location.

External image proxy

When displaying HTML email, all external images (when displayed) are proxied through the server. This prevents tracking and advertising cookies from being set in worker browsers, as well as protecting worker IP and location information. This will still ping “open/read” beacons when they are not filtered out. Previously, images were fetched directly in worker browsers.

The external image proxy can be configured by administrators from Setup » Mail » Incoming » HTML. This includes: the timeout for fetching images (default 2500ms), allowing/denying image redirects, the image blocklist, and the links allowlist.

Block email trackers

External images in HTML email can be filtered with an admin-configurable blocklist using a flexible rule syntax. This can block privacy-violating beacons, trackers, and advertisements before they’re displayed.

When displaying HTML email, clicking on external links now opens a redirect popup to confirm the destination. This helps combat “phishing” and other forms of deception where a link’s target doesn’t match its label. The redirect popup displays the main components of a link in a more human-readable format, including an SSL indicator, host, path, and query parameters. This provides a great expansion point for comparing links to databases of malicious hosts, displaying trust/reputation, showing a screenshot site preview, reporting phishing, etc. Previously, we relied on the vigilance of workers and their browsers.

External links in HTML email can be allowed with admin-configurable rules. This avoids the confirmation popup when clicking a trusted link, such as those within your corporate network, your team’s Cerb instance, or previously filtered URLs from a mail gateway.

Toggle between plaintext and HTML email

When viewing email messages, the display format can be toggled between plaintext and HTML right on the ticket profile.

Previously, if HTML was enabled, there wasn’t an easy way view the plaintext part (you could reply and look at the quoted text). [#110]

PGP Encryption

Encryption for everyone by default

Cerb no longer requires (or uses) the gnupg PHP extension for PGP email encryption, decryption, signing, or verifying. This makes it easier to use Cerb+PGP in more environments, including Windows.

The public keys from an existing gnupg keyring are automatically imported into Cerb during the update. However, due to limitations in the old gnupg PHP extension, private keys can not be exported. You’ll have to export your secret keys manually from the command line before deleting the keyring in ./storage/.gnupg/. [#995]

PGP public key records now store and display the ASCII-armored key text to simplify sharing.

We implemented our own keyring to provide the same functionality with fingerprints, id16, multiple user IDs (UIDs), and subkeys.

PGP public key worklists can now be filtered by UID, name, email, or any subkey fingerprint. Previously they could only be filtered by the primary fingerprint.

We improved the interoperability of Cerb’s PGP signatures with other email clients (e.g. Thunderbird/Enigmail). This included fixing an issue where our cleartext signature validation wasn’t including the raw un-decoded MIME part headers. We now also support signatures packets embedded in the encrypted PGP message (rather than cleartext). [#780]

Manage private keys

PGP private keys are now directly managed within Cerb by administrators.

Previously private keys had to be imported manually in the gnupg keyring which was very user-unfriendly.

Generate key pairs in the browser

PGP key-pairs can now be generated entirely in the browser. These support variable key lengths and multiple user IDs (UIDs), and follow the best practice of separate signing and encryption subkeys. This simplifies PGP setup for new users who don’t have an existing private key. It will also be useful when custom fields, portals, and bots need their own public/private keys for different processes.

Added encryption and signing to the mail toolbar

When composing and replying to email, new buttons for encryption and cryptographic identity signing are now available in the main toolbar. Previously, the encryption option was hidden near the bottom of compose/reply, and it was only visible if the gnupg extension was enabled. [#405]

Signing can be selected without encryption. When enabling encryption, signatures are also automatically enabled (but can optionally be disabled). Previously, messages were only signed when encrypted.

Verified signatures

When an inbound signed message could be verified by a stored public key, the message now includes detailed information about the signature, including the corresponding public and timestamp. Clicking on the signing fingerprint opens the public key’s card popup.

Previously, messages with a valid signature only displayed a check mark. The signature could have matched any public key in the keyring, lacking any identity verification.

Configure signing keys per group or bucket

PGP signing keys may now be configured per group or bucket. These don’t require the sender address to match a UID in the key.

Previously, the signing key was automatically selected based on the sender address of the group or bucket. This was cumbersome in environments with dozens or hundreds of sender addresses.

Passphrases on private keys

We now support passphrases on PGP private keys used for signing and decryption. Previously, the gnupg extension lacked this functionality.

Passphrases are stored encrypted in the Cerb database. However, we still recommend using non-master signing and encryption subkeys, since these are stored on a web server.

Filter messages by digital signatures

On message worklists, added signed.at: and signed.fingerprint: filters for cross-referencing signing public keys.

Records

Permanent logging of deleted records

Record deletions are now permanently logged in the activity log, with the record type, ID, and name. This provides an audit trail. [#1227]

Previously, some records logged a status change to ‘deleted’, but these log entries were eventually removed by maintenance after the record was removed.

Delete activity log entries

On activity log worklists, administrators may now delete entries. This makes it easier for an administrator to remove verified ‘login failed’ entries to clear an account lockout so a worker can attempt logging in again.

Improved performance on message worklists

Improved search performance when filtering messages by sender.

Portals

Usability

Fixed an issue with explore mode on community portal worklists.

Support Center improvements

  • In the Support Center portal, a user’s current password is now required when changing their password.

  • Fixed an issue in the knowledgebase on portals that prevented attached images from displaying properly.

Platform

PHP 7.4 support

Cerb and its dependencies are now fully compatible with PHP 7.4.

Audit logging

Improved POST request logging for auditing. Previously, most web server logs just recorded a request for POST /ajax.php in Cerb unless the request body was specifically being logged (not the default on most web servers). A _log query parameter is now included with information about the request (controller, action, ID). This doesn’t do anything on its own, but it makes it much easier to review logs.

Enhanced security for user-provided HTML content

Further secured the display of worker-provided HTML content throughout Cerb. This provides the same protection as HTML email when viewing external images and clicking links in: drafts, comments, sticky notes, sheets, widgets, bot interactions, email signatures, email templates, knowledgebase articles, and attachments.

This helps mitigate malicious workers, compromised accounts, and records created through the API (potentially from compromised credentials).

Updated dependencies

  • Updated the Twig template engine dependency from 1.33.2 to 3.0.2. [#1167]

  • Switched from the abandoned mtdowling/cron-expression library to dragonmantank/cron-expression.

Retired features

  • Retired the old ‘Simulator’ plugin that created sample tickets, orgs, and tasks. This is now better handled with packages.