As a DevOps specialist, I often find myself defending the value of documentation. It can often be a delicate topic in development teams, and individuals from different technical backgrounds and experiences can have fairly extreme opinions about it.
In this blog post I will take you through some history, the benefits of transparency and collaboration, how documentation must be easy to read, update and maintain, and debate whether it is actually required at all. Along with some guidelines for reducing the effort required, sizing, the importance of including manual steps, and how to keep the team motivated to maintain the documentation.
Documentation is necessary to explain how something complicated works, and also defines a critical process to ensure a successful software release.
We use outcomes, derived from discovery/research, stakeholder input and other business needs to define how and why we write code. I believe that our approach to writing documentation should learn from our approach to writing code.
Traditionally writing documentation was seen as a burden, listed at the end of a ticket’s acceptance criteria. It can often be seen by developers as uninteresting or not as important as writing the “actual code”. I have even heard it referred to as “not real work” by some programmers.
How do we go beyond this and ensure that documentation is maintainable, reliable and valuable?
A Dark History
Documentation obviously has a dark history in Software Development, appearing in the first few lines of the Agile Manifesto introduction, “Working software over comprehensive documentation.” Something in the rigid, linear, Waterfall development practices of the past have tainted this important part of development as the thing “that held up the delivery of software”. I have visions of software delivery being delayed by months by businesses demanding comprehensive documentation before shipping.
Traditionally the authoring of documentation was the final step before code can be released, but in modern agile practise, the inevitable delay to the release cycle isn’t tolerated.
Ironically everyone who has used a computer to write code has relied on other people’s documentation. The very first Unix Programmer’s Manual man page was written in 1971, and where would a lot of us be without this fairly rigid and unchanging process being adhered to over the last fifty years of computing, to enable us to use this complex toolset that our modern applications still rely on?
Transparent and Collaborative
Clearly everything in life cannot be tarred with the man page brush, and there are many different types of documentation required to suit each scenario. In every business, documentation is required for almost every aspect of the day to day running.
At Made Tech our Company Handbook is on Github, version controlled and Open Source. This was something that drew me to apply for a role at Made Tech, and I have heard similar positive feedback about this from many of my colleagues.
We have a business outcome to be open and transparent with our ways of working, job description and salary banding, and this Handbook fulfils that.
Documentation should meet user requirements
User-centred design should not just be focussed on delivering software aimed at users needs, but also ensuring documentation is aimed at the right level for the end-user or customer.
Often the poor people left running the application long after the development team have moved on to something else are an afterthought. Although, hopefully, with evolving DevOps practices, the team writing the application are also deploying and responsible for running it.
Therefore identifying your user, or group of users that your different documentation streams are for is paramount. Your user is likely to be yourself in a few months when you have completely forgotten everything about your own code. Be kind to your future self.
My code is the docs
In extreme programming circles, some would argue that the code should be self-documenting. This is a valid argument and at Made Tech we encourage the practice of TDD and Pair Programming, which both fall under the XP (Extreme Programming) label.
However, Martin Fowler states that code not needing documentation is a common misconception of extreme programming and, “Usually there is a need for further documentation to act as a supplement to the code.”
Writing clear, readable, well-structured code, with well-chosen Class, Method, Function, Variable names, and with descriptively named Unit Tests all help describe the code without the need for extensive code comments.
Many languages offer self-generated documentation from the code and features such as the Golang Test Framework Examples, which enables you to provide self-documenting implementation within your Unit Tests that can be published to the godoc website along with your package.
Ron Jeffries, one of the founding fathers of extreme programming, also argues that within a team the code should be self-documenting, and the team close and communicative enough that documentation is not required. However, “When it is time to hand off the software, whether to maintenance people or to users (perhaps programmers who will use the software to build other things, or other kinds of end users), then of course you need to build appropriate documentation to go with it. In the case of users, I’d think this would be just like any other kind of user documentation. In the case of maintenance programmers, we typically suggest a short document describing the system’s architecture or metaphor (perhaps with some UML diagrams), the key concepts, the important classes, and so on. This should, we suggest, be thought of as an introduction to the system, a high-level index if you will.”
Perhaps our outcomes for internal technical documentation should be to try and rely on the quality of the code. Documentation as code will meet some user needs, but not all, and this needs to be reviewed each time a piece of work is refined. Documentation should be treated as any other requirement and prioritised with other parts of the ticket.
Who are your users?
Establishing who our users are is paramount to ensuring that our documentation outcomes are met. At Made Tech we work heavily with Public Sector Government and Local Authority Departments. The GDS (Government Digital Service) Service Standards, are a set of guidelines around how to build digital services for the Government, built heavily around “User Needs”. This includes a set of Documentation Guidelines which help standardise documentation to end-users.
Documentation should be useful, valuable and have an outcome to ensure it fits our end-user needs. Nothing should be written just for the sake of it.
Learn from the community
Most complex community projects such as Kubernetes have contribution guidelines with clearly defined rules of engagement. It is clear that the users here can also be the maintainers themselves, and this self-managing cycle of information is version controlled in Github, peer-reviewed with pull requests with some fairly advanced automated pull request checks, allowing confidence to merge into master and deployed with minimal human interaction.
Using Github Markdown, the Hugo static website generator and deploying seamlessly with the click of a button via pull request checks and approvals via the CI/CD hosting platform Netlify. This process is impressive, and a lot of commercial products could take a page out of the Open Source community’s book here. I have seen similarly smooth processes done with Github Pages, S3 buckets and other tools.
If a documentation outcome could be to make it quick and easy to contribute, follow best practice in a version-controlled way, update and deploy a new copy, then this would be an excellent example.
Eliminating toil
Documentation can help eliminate toil. Toil is defined in Google’s SRE Handbook as Manual, Repetitive, Automatable, Tactical (reactive), leaves no enduring value after the work is complete, and scales linearly with traffic/load/users etc, then it is something worth investing in eliminating.
Handing over work with multiple manual steps in order to get something shipped is fine for an MVP or as a stepping stone towards full automation. No company I have ever worked with has prioritised full automated blue/green, feature flagged deployments, over having a viable product they can sell.
Clearly documenting the manual steps can feed back into your backlog of continuous improvement, and ensure all members of the team are aware of the more complex manual steps that may have manifested.
We can’t always go from zero to full CI/CD hero in the first iteration, but ensuring we set an outcome that we attempt to script and automate toil where we can, and document anything left over in a quick README to help our future selves.
Striving for Operational Excellence
Runbooks, Playbooks and Gamedays all feature heavily in the AWS Well-Architected Framework’s “Operational Excellence” pillar as being crucial documents, and feedback loops within a successful organisation. Those operating and managing the systems and specifically individuals “on-call”, will certainly see value in this area.
As DevOps engineers, we are increasingly responsible for building and maintaining our own code and systems. Being proactive in documenting manual processes in Runbooks allows us to improve and automate these processes, whilst Playbooks enable us to respond to failure swiftly. I like to hope that the newest member of a team should be able to deal with any kind of failure or outage to a certain extent without having to escalate to a more senior engineer.
Running regular Gamedays to simulate or preempt these events provides valuable insight and feedback into your Runbook and Playbook content through the process of a Root Cause Analysis (RCA) with the aim to stop the problem from happening again, whether it be through a code, process or cultural change by the team.
Minimum Viable Documentation
Documentation is essential for our end users to use or manage the software we deliver. Approaching the documentation process in the same Lean way as a Minimum Viable Product will reduce the size and complexity from the task.
Looking at process documentation outcomes, in the same way, breaks that cycle of it being a chore we do at the end of the project. Breaking the task down into smaller iterations with a fast feedback loop allows us to get down just enough information to be valuable, small enough to review and iterate on with minimal resources.
How much or little documentation can we get away with, how comprehensive does it need to be, and is it required at all? All of these choices will depend on the type of task or industry. Medical or aeronautical documentation where risk is high requires a high level of detail, audit and review. Whereas, other industries with more room for failure and iterative improvement can probably get away with a more minimal approach. This should be reviewed and tweaked for a balance between time taken to write, review and maintain, and the value it provides. A one size fits all approach certainly doesn’t apply here, and the outcome of the documentation will vary depending on the requirements.
Feedback Loop
One of the outcomes of technical documentation is that it should be a pleasure to read. Achieving this only comes from peer review and a feedback loop from your end-users. Pair programming is great for this but using a public version control system for your documentation if customer-facing allows direct feedback and suggestions from your actual end-users.
This kind of iterative feedback is invaluable. The development team demonstrating transparency and honesty by Open Sourcing your docs is always valued by your customers and end-users. Dogfooding your own documentation and manual steps after a period of time can be a good indication if they hold value for others.
Documentation standards
Why have an established standard? A hybrid blend of documentation locations like Confluence, Google Drive, Dropbox, Office 365, with a few repo level README.md’s thrown in is a recipe for disaster. We have all worked in places with multiple silos of documentation, or multiple copies of the same files in cloud storage, with names/timestamps/versions in the file name. When different teams and departments use different tooling, quickly things spiral out of control and soon enough you don’t know where to look for information, or where to put information.
Internally we have clearly defined protocols and tooling for certain tasks. However, at the start of customer engagements Made Tech will run a ways of working session, to align on many elements of the project and how we work and collaborate together. The outcomes from this session will establish, amongst other things, tooling choices, where documents are located, an agreed format and review period.
The ultimate outcome is referred to as the single point of truth.
Ensuring your outcomes have been met
Everything in software development needs a definition of done; however, in ongoing operational, process-driven departments, reviewing and updating documentation is a necessary workstream.
Documentation should only exist if valuable, well-written code with tests can often document itself at a technical level. Documentation should be user-driven with clearly defined outcomes before any work begins on it. In larger organisations and projects, some guidelines and frameworks can be useful to standardise and simplify a complicated but necessary process.
Documentation should be version controlled, peer-reviewed, easy to find and updated frequently, or removed if no longer relevant. Well written documentation can be a company asset, but done badly it can be “the thing” everyone hates and moans about.
Make sure you establish clear outcomes for your documentation before you start writing it, to ensure that it is not just acceptable, but valuable to its target audience.
