Technical debt, documentation, and runbooks

TL;DR:

• Run a quarterly audit of every custom integration (webhooks, apps, external services) using a structured checklist covering ownership, dependencies, tests, and monitoring.
• Every integration needs a README answering five questions: what it does, why it exists, how to deploy it, what Contentstack settings it depends on, and how to troubleshoot it.
• Document webhook routing in a single table so new developers see the full integration landscape without clicking through the Contentstack UI.
• Raise your bus factor above one: written docs, shared credentials (via secrets manager), code review, and onboarding walkthroughs.

Technical debt in CMS projects does not announce itself. Nobody files a ticket saying “our webhook handler is now unmaintainable” or “nobody knows why the SEO Sidebar Widget exists.” The debt accumulates entry by entry, dependency by dependency, undocumented decision by undocumented decision, until a routine change - updating a content type field, upgrading a framework version, onboarding a new developer - reveals how much implicit knowledge holds the system together. By then, the cost of paying down the debt is far higher than the cost of preventing it would have been.

This lesson covers how technical debt accumulates specifically in Contentstack projects, how to identify it before it becomes critical, what documentation actually matters (and what does not), and how to write operational runbooks that keep integrations running when the person who built them is unavailable.

How technical debt accumulates in Contentstack projects

Technical debt in CMS projects takes forms that differ from typical application development. In addition to the usual code-level debt (untested code, outdated dependencies, unclear naming), Contentstack projects accumulate debt in the content model, the integration layer, and the operational configuration.

Content type debt

Content types evolve. Fields are added for a campaign, used once, and never removed. A legacy_banner_text field sits in the Article content type with no validation, no documentation, and no entries that use it - but nobody deletes it because “something might depend on it.” Over time, content types accumulate deprecated fields that confuse editors and bloat API responses.

Content type debt also includes:

• Orphaned content types — content types that were created for a feature that was never launched or has since been retired. They still appear in the content type list, and editors may create entries against them.
• Inconsistent field naming — some content types use hero_image while others use banner_image or main_image for the same concept. No convention was established early, and each content type was built by a different developer.
• Over-broad content types — a “Page” content type with 40 fields that tries to represent every page on the site, when the content would be better served by four focused content types (Landing Page, Article Page, Product Page, Support Page).

Integration debt

Every custom integration (webhook handlers, Marketplace apps, external services) introduced in Module 6.1 creates potential debt:

• Webhook handlers nobody understands: A webhook handler deployed to a Lambda function 14 months ago. The developer who built it has moved to another project. The handler runs, but nobody knows exactly what it does, what content types trigger it, or what happens if it fails. The Contentstack webhook configuration says “Sync Handler” with no further description.
• Custom apps on unmaintained frameworks: A Marketplace app built with a framework version that is two major versions behind. The App SDK has released breaking changes. The app still works, but updating it requires a significant migration effort that keeps getting deferred.
• Hardcoded identifiers: A webhook handler that contains hardcoded stack UIDs, content type UIDs, or environment names. When the team creates a new environment or renames a content type, the handler silently stops processing the relevant events.
• Undocumented Automation Hub flows: Three Automation Hub automations that send notifications and sync data, created by different team members over the course of a year. Nobody has a complete inventory of what triggers what, and the automations have no naming convention that explains their purpose.

Configuration drift

Configuration drift occurs when the documented (or assumed) configuration diverges from the actual configuration:

• The README says the handler verifies against one webhook public key endpoint, but the deployed environment variable points to a different endpoint.
• The deployment documentation references a CI/CD pipeline that was replaced by a different pipeline three months ago.
• The content type documentation says the article content type has 12 fields, but it now has 18 because six fields were added without updating the documentation.

Identifying debt: the quarterly audit

Technical debt stays manageable only if you actively look for it. A quarterly audit of custom integrations - 30 to 60 minutes, with a structured checklist - prevents debt from compounding.

The audit checklist

For each custom integration (webhook handler, Marketplace app, external service):

Ownership:

• Who is responsible for maintaining this integration?
• If the responsible developer left tomorrow, could someone else take over?
• Is the owner documented in the integration's README?

Dependencies:

• Are all dependencies at their latest patch versions?
• Are any dependencies deprecated or abandoned?
• When was the last dependency update?
• Run npm audit (or equivalent) — are there known vulnerabilities?

Tests:

• Do the tests still pass?
• When were the tests last run?
• Do the tests cover the current behavior, or have features been added without test updates?

Deployment:

• Is the deployment process documented?
• Can a new team member deploy a change without asking the original developer?
• Is the deployment automated (CI/CD), or does it require manual steps?

Contentstack configuration:

• Does the webhook in Contentstack match the handler's expected events and content types?
• Has the content type schema changed since the integration was built? If fields were added or renamed, does the integration handle the new shape?
• Are the environment variables in the deployment environment correct and current?

Monitoring:

• Is there error alerting for this integration?
• Check the Contentstack webhook logs (Settings > Webhooks > [Webhook Name] > Logs) — are there delivery failures?
• Has anyone checked the monitoring dashboards in the last quarter?

Acting on audit findings

The audit produces a list of findings. Prioritize them by risk:

1. Security findings (vulnerable dependencies, leaked credentials): fix immediately.
2. Silent failures (webhook delivery failures nobody noticed, broken monitoring): fix within the current sprint.
3. Documentation gaps (missing READMEs, outdated deployment docs): schedule within the current quarter.
4. Technical improvements (dependency updates, code cleanup, test coverage): schedule when capacity allows.

Documentation that matters

Not all documentation is equally valuable. Writing extensive documentation that nobody reads is itself a form of waste. Focus on documentation that answers the questions people actually ask when they encounter your integration.

The integration README

Every custom integration - every webhook handler, every Marketplace app, every external service that talks to Contentstack - needs a README that answers five questions:

1. What does this do? One paragraph describing the integration's purpose and behavior.
2. Why does it exist? What business requirement or technical need prompted its creation. This is the context that disappears when the original developer leaves.
3. How do I deploy it? Step-by-step deployment instructions, including prerequisites and verification steps.
4. What Contentstack settings does it depend on? Which webhooks, content types, environments, tokens, or app installations must exist for it to work.
5. How do I troubleshoot it? Where to find logs, what common failures look like, and how to resolve them.

Content model documentation

Document your content model decisions - not just the schema (Contentstack's UI shows that), but the reasoning behind the schema:

# Content Model: Article

## Purpose
Represents editorial articles published on the blog. Used by the website,
mobile app, and newsletter systems.

## Key design decisions

### Why `author` is a Reference field, not a Group field
Authors are independent entities that appear on their own profile pages
and are referenced across multiple articles. Embedding author data inside
Article would create duplication and inconsistency. See Course 2, Module 2.2
on references vs. embedded data.

### Why `body` uses JSON RTE instead of Markdown
The editorial team needs inline image placement, embedded video blocks,
and structured callout boxes within article bodies. JSON RTE supports
these as custom blocks. Plain Markdown does not.

### Why `legacy_promo_banner` still exists
This field was added for a Q3 2024 campaign and is no longer used by any
frontend. It can be safely removed after verifying that no published entries
have data in this field. TODO: Remove in Q2 2025 cleanup.

### Fields consumed by external systems
- Algolia indexer reads: title, url, short_description, description
- Newsletter system reads: title, description, hero_image, author
- Mobile app reads: all fields except legacy_promo_banner

This documentation helps the next developer understand not just what the content model looks like but why it looks that way. Explicit notes about internal properties prevent a new developer from wasting time investigating a field that should have been removed.

Webhook routing documentation

When a project has multiple webhooks, document the complete routing picture:

# Webhook Routing

| Webhook name | Events | Content types | Target | Owner |
|---|---|---|---|---|
| Algolia Article Index | publish, unpublish | Article | algolia-indexer Lambda | @dev-team |
| Algolia Product Index | publish, unpublish | Product | algolia-indexer Lambda | @dev-team |
| Slack Content Notifications | workflow stage change | All | slack-notifier Lambda | @platform-team |
| CDN Cache Purge | publish | All | cache-purge Lambda | @infra-team |
| Newsletter Sync | publish | Article | newsletter-service API | @marketing-eng |

## Notes
- The Algolia Article and Product webhooks share the same Lambda function
  but use different handler routes (/webhooks/algolia-articles vs /webhooks/algolia-products).
- The CDN Cache Purge webhook fires for ALL content types including
  Navigation and Site Config. This is intentional - any published content
  change should invalidate cached pages.
- The Newsletter Sync webhook was added in March 2025. It only fires for
  articles published to the `production` environment.

Without this document, understanding the full webhook landscape requires clicking through every webhook in the Contentstack UI and then cross-referencing with deployed handler code.

Environment topology documentation

Document which environments serve which frontends and what tokens are in use:

# Environment Topology

| Environment | Purpose | Delivery token name | Consumers |
|---|---|---|---|
| development | Developer testing | dev-delivery-token | Local dev servers |
| staging | Pre-production QA | staging-delivery-token | staging.example.com |
| production | Live content | prod-delivery-token | www.example.com, mobile app |

## Token locations
- Delivery tokens are stored in the Vercel project environment variables
  for the website and in AWS Secrets Manager for the mobile app backend.
- Management tokens are stored in AWS Secrets Manager and used only by
  CI/CD pipelines and the content migration tool.

## Branch mapping
- `main` branch: serves production and staging environments.
- `redesign-2025` branch: feature branch for the 2025 redesign,
  serves development environment only.

Operational runbooks

A runbook is a step-by-step procedure for handling a specific operational scenario. Unlike documentation (which explains how things work), a runbook tells you exactly what to do when something goes wrong or when a routine task needs to be performed.

Runbook: Re-triggering a failed webhook

# Runbook: Re-trigger a Failed Webhook Delivery

## When to use
When the webhook logs show a failed delivery that needs to be retried
(e.g., the handler was temporarily down, and the event was not processed).

## Steps

1. Navigate to Settings > Webhooks in the Contentstack UI.
2. Click the webhook that failed (e.g., "Algolia Article Index").
3. Click the "Logs" tab.
4. Find the failed delivery in the log list (look for non-200 status codes).
5. Click the failed log entry to view the payload.
6. Copy the payload JSON.
7. Verify the handler is now running and healthy:
   - Check the health endpoint: curl https://api.example.com/health
   - Check error monitoring (Sentry/Datadog) for ongoing issues.
8. Manually replay the event using curl (including signature metadata headers):

   curl -X POST https://api.example.com/webhooks/algolia-article-index \
     -H "Content-Type: application/json" \
     -H "x-contentstack-request-signature: " \
     -H "x-contentstack-request-timestamp: " \
     -H "x-contentstack-request-version: " \
     -d ''

9. Verify the handler processed the event:
   - Check handler logs for the entry UID.
   - Verify the Algolia index contains the updated record.

## Escalation
If the handler continues to fail after manual retry, contact the
dev-team Slack channel (#contentstack-integrations).

Runbook: Reindex search after bulk publish

# Runbook: Reindex Algolia After Bulk Publish

## When to use
After a bulk publish operation (e.g., publishing 500 articles to a new
environment), where webhook-based indexing may have missed entries due
to rate limiting or handler capacity.

## Steps

1. Verify the bulk publish is complete:
   - Check the Contentstack publish queue (visible in the UI under the
     publish activity).
   - Wait until all entries show "Published" status.

2. Run the full reindex script:

   cd integrations/algolia-article-indexer
   NEXT_PUBLIC_CONTENTSTACK_API_KEY=blt... \
   NEXT_PUBLIC_CONTENTSTACK_DELIVERY_TOKEN=cs... \
   ALGOLIA_APP_ID=... \
   ALGOLIA_ADMIN_API_KEY=... \
   node scripts/full-reindex.js --content-type article --environment production

3. The script fetches all published articles from the Contentstack
   Delivery API and indexes them in Algolia. Progress is logged to stdout.

4. After completion, verify the index:
   - Check the Algolia dashboard for the expected number of records.
   - Search for a recently published article to confirm it appears.

## Notes
- The full reindex script respects Contentstack API rate limits
  (for example, 200 requests/second on many paid CDA plans, or 1000 requests/minute on free plans).
- Estimated time for 1,000 articles: ~5 minutes.
- The script is idempotent - running it multiple times is safe.

Runbook: Deploy a new version of a Marketplace app

# Runbook: Deploy Updated Marketplace App

## When to use
When a new version of a Marketplace app (e.g., PIM Product Selector)
needs to be deployed to production.

## Pre-deployment checklist
- [ ] All tests pass locally: npm test
- [ ] App tested in development stack (installed and functional)
- [ ] App SDK version compatible with current Contentstack platform
- [ ] No breaking changes to the Custom Field data format

## Steps

1. Build the production bundle:
   cd apps/pim-product-selector
   npm run build

2. Deploy to hosting (Vercel):
   vercel --prod

3. Verify deployment:
   - Open the deployed URL in a browser: https://pim-selector.example.com
   - The app should display a loading state (it needs Contentstack host
     context to fully initialize).

4. Test in the Contentstack UI:
   - Open an entry that uses the PIM Product Selector custom field.
   - Verify the field loads and displays the correct UI.
   - If a product was previously selected, verify it still displays.
   - Test searching for and selecting a new product.
   - Save the entry and verify the field data persists.

5. No changes needed in Developer Hub unless the app's locations,
   scopes, or URLs have changed.

## Rollback
If the deployment introduces issues:
   vercel rollback
This reverts to the previous deployment. The app's iframe loads
the hosted URL, so the rollback is immediate - no Contentstack
configuration changes needed.

The bus factor

The “bus factor” is the number of people who would need to be unavailable before a system becomes unmaintainable. For many Contentstack integrations, the bus factor is one - a single developer who built the integration, knows where it is deployed, understands its configuration, and has the credentials to modify it.

Increasing the bus factor does not require pairing on every task. It requires:

• Written documentation (the README, runbooks, and model documentation described above).
• Shared access to deployment pipelines, monitoring dashboards, and credentials (via a secrets manager, not shared passwords).
• Code review for all integration changes, so at least one other developer has seen the code.
• Onboarding walkthrough when a new team member joins - a 30-minute tour of the integration landscape, pointing to the documentation.

If the developer who built the Algolia webhook handler were unavailable tomorrow, could someone else debug a delivery failure, deploy a fix, and verify the index? If the answer is no, your bus factor is one, and the documentation gaps are the highest-priority debt on your backlog.

Common mistakes

Common pitfall:

Treating technical debt as something to fix “when we have time” guarantees it compounds. A field that should have been removed six months ago now has entries using it by accident. A dependency two versions behind now requires a breaking migration. Allocate explicit, recurring capacity — one hour per week, one day per sprint, or a quarterly cleanup day — and protect that time.

1. Documenting everything at the wrong level of detail. A 50-page document that explains every line of code is as useless as no documentation. Document the why, the how-to-deploy, and the how-to-troubleshoot. The code explains the what. Focus documentation on information that cannot be derived from reading the source.
2. Treating technical debt as something to fix “when we have time.” Teams never have time. Debt compounds. A field that should have been removed six months ago now has entries using it by accident. A dependency that should have been updated two versions ago now requires a migration. Allocate explicit capacity for debt reduction - one hour per week, one day per sprint, or a quarterly cleanup day.
3. Assuming Contentstack's UI is sufficient documentation for webhook routing. The Contentstack UI shows individual webhook configurations, but it does not provide an overview of how all webhooks, Automation Hub flows, and external integrations interact. The routing documentation described above fills this gap and is the first thing a new developer should read.