Generative AI and ChatGPT have created a lot of buzz lately, sparking discussions about how and if this technology can replace other tasks or jobs. One field where this topic arises is that of internal documentation for software teams. Can generative AI replace the need for creating documentation altogether?

There are a few directions to explore here. The first involves removing the need for documentation altogether, where AI would help explain the code or task at hand. Secondly, AI has the ability to create and maintain accurate and up-to-date documentation automatically, which reduces the burden on software teams who must manually update docs. 

Additionally, generative AI can use natural language processing to generate documentation that is easy for developers to understand, making knowledge about code more accessible to team members. 

However, there are also potential drawbacks to using generative AI for internal documentation, such as the need for significant training data and the potential for errors in the content generated.

Removing the need for documentation altogether

Since its launch in November last year, OpenAI’s generative AI tool ChatGPT has taken the internet by storm and amazed users with its abilities.

“The striking thing about the reaction to ChatGPT is not just the number of people who are blown away by it, but who they are. These are not people who get excited by every shiny new thing. Clearly something big is happening.” Paul Graham

Beyond the technical excitement, many people started to speculate about the implications of new AI models, and whether they may be able to replace writers, programmers, or other jobs altogether.

For example, OpenAI’s models have done an extraordinary job generating and manipulating code to completion, not limited to: 

• Turning comments into code
• Completing your next line or function in context
• Bring knowledge to you, such as finding a useful library or API call for an application
• Add comments
• Rewriting code for efficiency

What does this mean for internal documentation? Despite its importance, documentation is often neglected as the process of creating and maintaining it is painful. That begs us to ask ourselves if generative AI will replace the need to create documentation?

One approach would be to stop relying on documentation altogether. Instead, developers could use AI to explain existing code in case it is not clear. While it sounds promising, we should be very cautious with adopting such a mindset.

One vital role of documentation is to describe the business logic behind the codebase/service. This knowledge cannot be extracted from code, no matter how sophisticated the model analyzing the code is. 

Another important aspect of documentation is explaining design decisions. Understanding the idea behind the current design may be extremely helpful for an engineer trying to get hold of the current state of things. Moreover, important design documents often explain why a certain design was not chosen as a solution, in which case it will obviously not exist in the code. So no AI model would explain with certainty why a design was not chosen.

There are some aspects of knowledge about software that cannot be extracted from the source code, so generative AI couldn’t just replace them. However, there are ways where generative AI can be extremely useful.

Using AI to maintain accurate and up-to-date documentation

In some cases, generative AI can definitely help by generating documentation on-the-fly, even if it doesn’t exist. The best example would be documentation strings for functions. ChatGPT, for example, does a great job generating docstrings. 

For a developer, it may be easier to read those in natural language rather than the code itself. Furthermore, human-created docstrings may be incomplete or outdated, whereas generative AI can make sure every parameter of the function is documented, and generate documentation every time the code changes. As long as the model can accurately explain the code, this is a useful tool.

Using generative AI for documentation that is easy to understand

Generative AI models already do a great job at creating introductions, summarizing documents, and helping developers structure their documents by suggesting sections and so on. With these capabilities, AI can help engineers create documents that are easy to understand. Moreover, AI models help engineers read documents by summarizing them and rephrasing sentences.

In addition, AI models can help create documents with grammatically correct and coherent language. Not all engineers enjoy writing, and not all are adept at it. If English isn’t your dominant language, writing technical documents may be daunting. AI models are already embedded in tools and products to help engineers express themselves, and with generative AI models, it may be even easier to interact with them.

AI models can also reflect on the code, and help engineers create the desired documents. A model could, for instance, identify best practices that are common within the repository, and suggest mentioning these practices.

Generative AI drawbacks for internal documentation

Earlier we discussed the option of replacing documentation altogether with generative AI, and concluded it would be problematic as important information is fundamentally missing from the code.

Generative AI models are already very good at explaining code snippets in isolation. Yet, this is not enough. Developers today are faced with trying to understand the full picture of the code. The flows of code can be tricky to understand, especially when they span across multiple repositories or external systems. These flows are not only hard to understand for an engineer, but also for any algorithm. A big reason comes back to your architecture, where not all relevant data required to understand the entire flow may be available, as every step doesn’t exist within the codebase.

When creating documentation, we should keep in mind what it strives to accomplish – transform knowledge to the minds of those contributing to or modifying the source code. Generative AI is great at providing some of the relevant information, but not all.

Beyond creating documentation

Creating documents is only the first step. To get value from the documents, they need to be up to date, and developers need to find the relevant documents when needed.

Even if an engineering team decides to fully rely on generative AI to assist or create document drafts, as the code evolves – it will quickly become obsolete or misleading. To keep documentation up to date, other tools are required.

For example, with Swimm, developers create walkthrough documents – engaging cross-repository docs with content components like variables, tokens, file/folder paths, code snippets, and diagrams. These interact directly with the code and are automatically synchronized as changes are made. 

Documentation is kept up to date using patented AI that makes recommendations as part of CI/CD workflow. On every PR, the algorithms indicate when everything is okay, alert you when something needs reviewing, give you corrections to approve, and can even automatically correct simple issues for you. This way, the documentation is always up to date, and developers can trust it.

Also, by using Swimm’s IDE plugins, developers don’t have to actively search for documentation. Relevant documents find them. While working within the IDE, and reading through code, Swimm’s plugin shows a comment from a relevant document that may be helpful for this part of the code. Upon clicking on the comment, the document opens right from the IDE. In addition to the “classic” ways of indexing documents by tagging or organizing them within folders, this new way of finding documentation makes sure you really have the docs as you need them.

Reflections

Generative AI is here to stay and will disrupt many industries. In this post, we considered its possible implications for internal documentation.

Let us summarize the main conclusions with a table:

Generative AI poses an amazing opportunity for many fields, and the ability to create high-quality software documentation, specifically. By utilizing its potential while understanding its limitations with other tools, we can form a new, more efficient reality.