Automatic Updates
One of dope fundamentals is to reduce operational overhead & manual work, including deploying new agent versions, and waiting for policy updates to occur. Instead, an admin can instantly push new policies to an endpoint, and the endpoint will automatically upgrade to the latest version.
Our number one focus for this process is to be invisible to end users, by rigorously testing our releases, including build acceptance tests, automation QA, manual sanity tests, stress/load tests, and internal -> phased rollouts.
For transparency, we've documented frequently asked questions around how updates are pushed to your device and how we ensure the process has no impact to your end devices.
What different updates can occur with Dope?
All dope upgrades are invisible, they are designed to be unnoticeable by end-users. Software updates do not require any system restarts.
dope.endpoint: this is the main installable on your device (MacOS/Windows). It consists of multiple components, including: - Redirector: re-routes traffic to the Proxy. On Windows, this is driver that uses Microsoft's WFP to operate. On Mac, this is a network extension that uses exposed Apple's APIs. - Proxy: performs the SSL inspection, URL filtering, etc. The code is identical in both OS
Config/Policy Update: these are admin-configured changes, such as blocking a web category, or adding a new URL to a custom category, adding a new domain to the cloud app controls etc.
Global Application/Domain Bypass: these are delivered via the policy update, but are a set of dope-specified applications that are known to cause issues with SSL Inspection proxies. For example, the Zoom application
URL Categorization: these are requested on-demand by the endpoint and are kept up-to-date via a regular cache update. For example, if a user visits google.com, the categorization for google.com will be requested, locally cached, and updated regularly
Each update occurs slightly differently:
dope.endpoint: when a software update becomes available for a specific device, it is advertised to the endpoint via the configuration. The endpoint software update is then downloaded, and installed via the updater process. Except during a rollout period, all devices should be on the latest version
Config/Policy Update & Global Application/Domain Bypass: both of these updates are part of the policy. So, if an admin makes a SWG policy change, or if a new application is added to the default bypass, the endpoints will retrieve the latest policy - Policy Push Websocket: the endpoint websocket receives notifications that a policy update has been made. In real-time, the endpoint will call the API to retrieve the latest policy update - Polling Policy Refresh: the endpoint automatically calls the API to retrieve the latest policy multiple times an hour, or when a device comes back online
URL Categorization: these are either requested on-demand as websites are accessed, or updated as part of the category cache. URLs are tiered to ensure that requests are made sooner for websites that might change categorization. In the cloud, these are automatically updated multiple times a day.
How are updates tested?
Each area of the different updates has a varied level of testing:
dope.endpoint Testing
Each release is thoroughly tested in a multi-step process, as detailed later, released in a multi-stage process.
BAT (Branch Build Acceptance Test)
To allow for developers after making a Pull Request (could contain feature additions, bug fixes, or enhancements) to automatically compile a branch build and test against the top use-cases an end-user will go through. The BAT includes SSL inspection test-cases, bypass list, category blocking, etc.
Manual Sanity Test
Builds are manually tested to ensure there are no easy-to-spot issues to be fixed
Regression Test
Takes RC (release candidate) build and runs through an automated regression test suite that contains a very large set of test cases which includes feature coverage, previous bug coverage, and build upgrade coverage
Compatibility Test
Takes RC build and runs through compatibility tests with all major VPN clients, Endpoint Security software, and various networking environments (IPV6/V4 hybrid)
Load/Stress Test
Takes a fully tested RC build and runs the build through significant load for multiple days while monitoring for memory leaks, crashes, CPU spikes, and other potential issues
After the testing is complete and the release is ready, it is released internally on employee devices.
Config/Policy Update Testing
Admins push policy updates live via the cloud console. This is not controlled by dope.security. The general process of receiving a policy, whether via policy push or polling policy refresh is regularly tested via automation outlined above
Global Application/Domain Bypass Testing
Any addition to the default bypass list is manually tested before being pushed live.
URL Categorization Testing
URL categorizations are tested as part of all tests and is not tested specifically (except for uptime). The main risk is a mis-categorization of a website, in which case we focus on speed of remediation/rollback of an incorrect categorization.
Does dope.security use dope.security internally? (dogfood)
All devices at dope.security run our endpoint to ensure that we can see & resolve any issues that could've sneaked through release testing. The version in our environment is typically the upcoming release as a test-bed. It is in two phases:
Internal Environment: devices that are on branch builds, automation & load systems
Production Environment: employee devices that are on release candidate versions, that can eventually be used for a customer release
Both the Internal & Production environment both run the latest cloud deployment to reduce edge-cases
How are updates rolled out?
The multi-stage rollout process helps ensure there are no large scale surprise issues, especially in customer environments. The release phase applies to the dope.endpoint only. The dope.cloud_console only uses feature flags for releases.
Internal/Prod Dogfood
Allows for sanity tests and ensuring that no issues are seen on internal employee laptops
Partner & Early Access Tenants
Certain partners & customers have opted to receive releases at the earliest point. After deployment, these are monitored to ensure devices are healthy.
Customer-Specified Endpoints
Certain customers have specified canary endpoints that should receive builds first to ensure that if there are any issues it is caught early
Customer Tenants
After all health checks are positive, customer tenants are progressively updated to use the latest build
Does dope monitor for failures, and rollback?
As upgrades are rolled out for each area, there are different options available to rollback in case of any failures. Each upgrade phase is monitored for failures.
dope.endpoint: If a failure requiring rollback is required, the last release is prepared and all tenants are updated to use the last release (at an up-to-date version). If an individual endpoint upgrade failure occurs, rollback to the last version occurs automatically.
URL Categorization: If any URLs are incorrectly categorized, the categorization can be corrected and will reflect in different time periods on a device based on the scenario: - Suspicious/Malicious False Positive (Tier 3): categorization expires after one hour, device will retrieve corrected categorization as soon as it is updated, or not more than one hour later - Popular Domain Incorrect Categorization (Tier 1): if a highly popular domain is incorrectly categorized, such as facebook.com, our team will push a domain bypass to immediately take effect on devices, or when the device comes back online
Last updated