TLDR: Here is a summary
As the world is rapidly adopting cloud platforms among the world, organization’s dependency on additional SaaS, PaaS and IaaS vendors increases. The need for cyber security in these cloud-native contexts also arises because of this. When we look at for example isc2’s CCSP (Cloud Certified Security Professional) certificate they put forward IRM, DLP and SIEM as key tools to track, control and monitor data in the cloud. In this Article I want to zoom in and highlight a series of challenges regarding the monitoring of these solutions using the SIEM as a method for securing the cloud.
SIEM’s can ingest logs from many different sources for threat detection and in the cloud context these sources will primarily come in API log integrations (streaming or bulk). The problem comes in if one’s organization already has many cloud native applications, has over a 100/1000+ cloud-native SaaS services each with their own API’s and your goal is to onboard these log sources into your SIEM tool of choice. The following challenges try to outline the major issues with all these API integrations and propose potential solutions for API developers of these cloud services to provide an increased level of satisfaction with their customers and an overall increased security posture in the cloud landscape.
Critical Note 1: This is the current status as I have experienced it in the cloud landscape, this might or might not rapidly mature overtime and in the future these challenges might be less of an issue.
Critical Note 2: The cloud landscape is a wide landscape with many different vendors ranging from high and low maturity therefore these challenges might or might not apply for some of them depending on situation.
Critical Note 3: Some of these items are not new challenges but general challenges with any shape or form of log collection.
1. API Log Collection Endpoints are Sometimes Not Available
Most cloud native apps that are created generally are built from the “API-centric” philosophy unfortunately these API endpoints are generally focused on application functionality and generally do not provide an audit/security logs retrieval API endpoint.
Recommendation: Create an API endpoint for audit/security logs within your cloud service.
2. API Log Collection is Barely Real-Time and Sometimes Even Very Delayed
Some cloud vendors provider an API endpoint for log collection but when configuring this on your SIEM of choice, one might be hesitant to pull every 30 seconds logs from this endpoint and moves it to 5 minutes in order to reduce overloading/trigger rate limiting on the API. Besides that, once the log is pulled in some vendors might have a back-end delay ranging from a couple of minutes to one or many hours when it comes to piping these events into the API endpoint this is very troubling if one wants to do real-time or near real-time threat detection with a SIEM.
Recommendation: Create an audit/log API endpoint that provides logs in (near) real-time.
3. API Log Collection Availability is Rarely Committed in Cloud-Centric Vendor Contracts
Generally, a cloud service comes with an SLA that agrees on the availability of the service provided in terms of percentages, unfortunately it is all too common this is not scoped in for API endpoints that provide audit/log capabilities although this might be critical for some customers due to reporting or compliance requirements. This allows cloud vendors to have long term outage or extreme log delivery delay on an audit/log endpoint and not be accountable or penalized in terms of the contractual context of the cloud service.
Recommendation: Commit to API audit/log endpoint availability metrics inside the SLA.
4. API Log Collection Audit/Log Data is Sometimes Limited or Unusable for Threat Detection
In cases where the API endpoint is in place for audit/log collection and events are coming in, sometimes the API is not developed and key fields used for threat detection might be missing, for example: Source IP, Account name, UserAgent, Endpoint called and Action executed. This renders the onboarding initiative for the Audit/Logs through the API as a non-productive activity.
Recommendation: When creating an audit/log Endpoint provide as much useful data as possible.
5. API Log Collection Sometimes is an Additional Paid Service
There are cloud vendors out there that require the customer to pay additional money and subscribe to an additional subscription in order to provide their API audit/log endpoint access to the customer. This is very annoying if you are trying to onboard many different cloud apps. Many of these initiatives stop because of a commercial challenge involved causing a blind-spot in security monitoring the specific cloud service in terms of security monitoring. In my opinion If it’s your own data you should be able to access it without additional cost, this is just like charging for an additional napkin inside a restaurant.
Recommendation: Provide audit/log API endpoint as part of the standard package.
6. API Log Collection Endpoints for “Transparency Logs” are very uncommon among vendors
Transparency logs pertain to the backend operators that access your data, transparency logs might provide critical insight into the behavior of the cloud vendor as these might look into your sensitive data for no real-reason beyond the administration of the cloud service. This is still a relatively new thing for most cloud vendors and therefore is mostly absent from most API endpoint offerings.
Recommendation: Provide a “Transparency Logs” API endpoint as part of the standard package.
7. API Log Collection Tokens Do Not have a Dedicated “Audit Log Viewer” role
When onboarding an API log sources inside the data collection tool like a SIEM you are generally required to use an API token for authentication, along with this API token there is also an authentication role associated that provides the access to the specific endpoint required. Unfortunately, it is all too common that a token needs a “full admin” authorization role to just access the basic audit/log API endpoint, creating an additional risk when accidentally leaking the key.
Recommendation: Provide a dedicated “Audit Log Viewer” authentication role for log collection.
8. API Log Collection Delivery Methods Vary and Might Have Rate Limiting or Hard Limits in Place.
There are a variety of API log collection methods in place each with their pro’s and con’s, generally the decision is made based on what the cloud vendors application context is. Collection mechanism might be for example a: “streaming API” (streaming events in near real-time), there is a “streaming API” with a backlog function (halts every half hour and then clears the backlog in order to continue streaming again), “Bulk (or batch) API” every period a batch of logs are pulled or pushed using a established or scheduled a API call. On top of that rate limiting might be in place to limit events per API call or overall hard limits on how many events can be pushed out over a period of time.
Recommendation: Provide dashboarding and configuration options on API usage and limits.
9. API Log Collection Methods are Constantly Changed and are Not Standardized
Although most API log collection endpoints are using a restful JSON based API and most of them are speaking the same protocol, every interaction still varies vendor per vendor depending on authentication schema’s, delivery method’s, API rate limiting and other challenges mentioned before requires every Security Monitoring vendor to create their own Integration code for every cloud vendor causing a lot of work of tracking and integrating every new cloud vendor’s API’s for log integration. Sometimes these API endpoints are modified without informing the end-user or completely deprecated without announcement causing the endless loop of trying to catch up with cloud vendors on their log collection services. There is no global standard (as far as I have seen) regarding a standardized API endpoint log collection pattern that is replicated across all vendors easily with the likes of let’s say “syslog”.
Recommendation: Comply to an API industry-standard and inform users timely when changed.
10. API Log Collection Audit Log Sometimes Has 1 Array with Highly Dynamic Bulk Data Population
In some cases, an API endpoint provides all the things we need as a Security Monitoring professional but due to the large amount of data an application produces many different “columns” of data is pushed into a key value pair series into 1 JSON array. On top of that these columns might vary per events causing even more additional challenges in terms of normalization of these logs when parsing the JSON events to a readable format. In traditional log collection, using a CEF (Common Event Format) was generally the solution for these dynamic fields: “devicecustomstring1, 2, 3” etc. Unfortunately, this is not used commonly within API endpoints from the start.
Recommendation: Split data using a standardized log format in separate arrays, not bulk populate.