In software, auditing means tracking user or system activities for various needs, such as business or security. An example would be -
user X tried to access resource Y.
When I encuntered issues with auditing at my current company, I looked for solutions online, most of which, were either vague, lackluster or plain simple. That is why, after having implemented and dealt with auditing at scale, I would like to share my thoughts. In this post, I will show the various methods for implementing auditing together with code examples and pros and cons.
All code examples will be written in Golang.
The simplest form of audit is to log the event that happened from the business logic.
Afterwards, it is possible to aggregate logs and send them to an ELK service for parsing and viewing.
- Easy to implement
- Easy to ship to any 3rd party service
- Can take a while until the logs are scraped, parsed and shipped
- Writing many operations to stdout will cause a performance hit
Developers will usually have some sort of database like MongoDB or PostgreSQL that they use, which can be used for audits as well. It is also possible to have a separate database for the auditing.
- Main application db can be used for the auditing as well, if the service’s scale is low, which saves in operational costs and maintainance
- Audits can be exposed as an API for customers, marketing and various teams in the organization.
- Not scalable if you use the same database as your application
- Tougher to manage in a microservices environment as each service should have its own database, meaning no single owner of the audit data, unless you make an audit service, which we will discuss later in the post.
Have a single service for auditing, which is responsible for managing all audit data. All other services can communicate with it either synchronously (HTTP) or asynchronously (Publish/Subscribe, Queues, gRPC).
- Easy to incorporate to an existing architecture.
- New audits must be explicitly created with a call to audit service, unlike the implicit nature of Event Sourcing
From microservices.io [1, 2]
Event sourcing persists the state of a business entity such an Order or a Customer as a sequence of state-changing events. Whenever the state of a business entity changes, a new event is appended to the list of events. Since saving an event is a single operation, it is inherently atomic. The application reconstructs an entity’s current state by replaying the events.
CQRS - Command Query Responsibility Segregation 
At its heart is the notion that you can use a different model to update information than the model you use to read information
To keep things simple, with event sourcing, your business logic fires commands that in turn generate events that are appended to the store (append only database that is usually fast for writes), and each time a new event is appended, it is also published for the appropriate consumers to react.
Event sourcing goes hand in hand with CQRS, due to events needing some sort of snapshot (a normalized view), to allow for fast reads. If we were to query only from the append only store, we would need to start from a known state, extract all relevant events and apply each one of them to the known state, which is very slow. With CQRS, we store a snapshot of the latest data and when a query comes, we are able to return the snapshot instead of reconstructing the state.
Back to our example, we fire the
AddUserToGroup Command, which generates the
UserAddedToGroup event. Afterwards, the group consumer receives the
UserAddedToGroup event, and reacts accordingly by populating the new data in a normalized way for easier querying.
Groups service command side
Group service query side
- We get audit out of the box for all operations
- We are asynchronous from the get go, which can help with scale issues down the line
- The event store is difficult to query since it requires typical queries to reconstruct the state of the business entities, unless we ALSO save the data in a normalized way (CQRS), like in our group handler example. This increases operational costs as well as adds complexity to the developers.
- Unfamiliar programming style for most developers
|Audit Type||Complexity||Scfalablity||Ease Of Integration With Existing Architecture|
Each of the implementations above has its pros and cons and you should always start with the simpler solution that can be implemented with as little effort as possible. If you are lucky enough to grow with your company to larger business needs and scale, you should consider the more scalable approaches, which are also more challenging. Regardless of what you choose, always try to write your code modular and consistent, like I explained in my previous posts, Ports and Adapters  and Clean Architecture .