DevRev Post Incident Analysis (PIA) - INC-2 - Ticket owner field fails to load with “something went wrong” TypeError originId undefined

Summary
Pebl users reported that the ticket owner field fails to load and displays a "something went wrong" message. The issue was identified as a TypeError: Cannot read properties of undefined (reading 'originId'), which began around 12:30 PM UTC-3. The incident was acknowledged and mitigated after investigation and a hotfix was applied.

Timeline

• 12:30 PM UTC-3: Users report the owner field not loading on tickets.

• 12:31 PM UTC-3: Incident owner assigned and issue identified as a front-end problem in the Works UI.

• 12:32 PM UTC-3: Confirmation that the issue affects the ticket detail UI.

• 12:44 PM UTC-3: Ticket ID TKT-1796143 identified as affected.

• 1:07 PM UTC-3: Users report that the owner field fails to render consistently.

• 1:26 PM UTC-3: New reports from multiple users indicate widespread impact.

• 1:55 PM UTC-3: Investigation reveals a potential link to a recent deployment.

• 2:15 PM UTC-3: Incident declared officially.

• 4:51 AM UTC-3 (next day): Root cause identified as accessing properties of an undefined service account.

• 5:45 AM UTC-3: Incident was mitigated with hotfix applied to address the issue.

• 7:37 AM UTC-3: Incident object stage updated to "mitigated" after confirming with customers.

Analysis

1. How was the incident detected (alarm, manual, etc.)?

Manual — customer-reported. Users hit a hard render crash (blank/broken UI) in user pickers, selectors, field cells, and mentions. The console error TypeError: Cannot read properties of undefined (reading 'originId') was captured, and HAR files were collected from affected sessions to aid diagnosis. No automated alert fired.

2. How was the root cause diagnosed?

By correlating the error string with source. Because minifiers don't rename object property accesses, the crashing .originId read had to be one of only 4 source sites that read that field. The HARs showed service-accounts.get returning a valid service_account payload, ruling out a bad API response and pointing to a client-side cache-shape collision: the React Query key serviceAccountKeys.detail(id) is written with two incompatible shapes — useDLServiceAccount stores the adapted { serviceAccount }, while useDLListServiceAccountsById (used by pickers/fields) writes the raw Axios response ({ data, status, headers }, no .serviceAccount). The readers used response?.serviceAccount.originId, guarding the response object but not the inner .serviceAccount, so a poisoned-cache read threw. Confirmed with a unit test that feeds the raw-Axios shape and reproduces the exact error deterministically.

3. Was there a backlog item that could have prevented the issue?

Yes. The PRE team reported this exact issue during a regression test run (TKT-103519). The defect was known prior to deployment.

4. Why was it not completed before the incident?

Multiple team members were unable to reproduce the issue locally, so it was not treated as a deployment blocker. The change shipped to production despite the open regression report.

Five Whys

Problem (as seen by the customer): The app crashed (blank/broken UI) when opening user pickers/selectors, viewing field cells, or rendering mentions.

1. Why? A React render threw Cannot read properties of undefined (reading 'originId'), which unwound the component tree and blanked the UI.

2. Why? ServiceAccountAvatar unconditionally computed const originId = response?.serviceAccount.originId on every render. The optional chain guarded the response object but not the inner .serviceAccount, which was undefined.

3. Why? The value read back from the serviceAccountKeys.detail(id) cache was the raw Axios response (no .serviceAccount). React Query serves cached data even to disabled queries, so the crash occurred for all affected users regardless of any configuration.

4. Why? Two hooks write that same query key with different shapes: useDLServiceAccount stores the adapted { serviceAccount }, but useDLListServiceAccountsById stores the raw Axios response. Pickers/fields call the list hook and commonly seed the cache before an avatar reads it, so "last writer wins" left the wrong shape in cache.

5. Why? There is no enforcement — at the type level, lint level, or test level — that all writers of a given React Query cache key must produce the same contract/shape. Without such a constraint, two hooks silently diverged on one key, and the new originId readers were added without accounting for the pre-existing dual-shape write.

Customer Impact

Impact was broad. Any affected user crashed when a ServiceAccountAvatar rendered for an id whose detail-cache entry had been poisoned by a picker/field (useDLListServiceAccountsById). Because the originId read is unconditional and React Query serves cached data to disabled queries, the crash spanned user pickers, user/owner selectors, field cells, and mentions — high-traffic surfaces used across the product. Exact scope is hard to quantify: there was no monitor to size it, so impact was inferred from support/customer report volume rather than measured. That absence of an error-rate monitor is itself a key gap (see Action Items).

Lessons Learned

How could we have detected this in half the time?

The issue was already detected — the PRE team caught it in regression testing and filed TKT-103519. We should have blocked the deployment when a regression defect of this severity was raised in QA. A stricter release gate that treats unresolved regression failures as deployment blockers would have prevented this from reaching production entirely.

How could we have mitigated this in half the time?

Given the broad customer impact, we should have immediately rolled back the production deployment rather than attempting a forward-fix. The time spent diagnosing and preparing a hotfix while users continued to crash was avoidable — a rollback would have restored service in minutes while the team worked on the fix offline.

What systemic gap allowed this to happen?

There is no enforcement that a single React Query cache key can hold only one contract/shape. We need a mechanism — whether via types, lint rules, or runtime validation — that prevents multiple hooks from writing incompatible shapes to the same cache key. Without this, any new reader of a shared key is at risk of hitting a poisoned-cache shape written by an unrelated hook.

Action Items

Reduce Customer Impact

• TKT-104013 — Root-cause fix: make useDLListServiceAccountsById store the adapted { serviceAccount } shape (or use a distinct query key) so serviceAccountKeys.detail(id) is never written with the raw Axios response. Add a shared/typed contract or test so writers of the same key can't diverge again. (Work starting next.) Also Add an error boundary around service-account avatar/picker render slots so a bad data shape fails soft to a fallback avatar instead of crashing the tree.

• TKT-104943 — Add React Query cache key contract enforcement: implement a type-level or lint-level mechanism that ensures all writers of a given query key produce the same shape, preventing silent dual-shape collisions.