The 75% Trust Deficit
Seventy-five percent of executives don’t trust the data in their IT reports. Forrester research quantifies what recipients of MSP reports already feel: the numbers don’t tell the truth they pretend to tell.
The distrust is earned. Reports designed to demonstrate success demonstrate success. They don’t reveal problems, near-misses, or degradation trends. The green dashboard becomes a confidence trap.
The Vanity Metric Problem
“Vanity metrics” measure what’s easy to count, not what matters. They look impressive without informing decisions.
| Vanity Metric | What It Appears to Show | What It Actually Shows |
|---|---|---|
| 99.9% server uptime | System reliability | Server ran, says nothing about application |
| 500 tickets closed | Productivity | Volume processed, not value delivered |
| 15-minute average response | Attentive service | Fast acknowledgment, not fast resolution |
| Zero security breaches | Strong security | Nothing detected, possibly nothing detected |
| 100% backup completion | Protected data | Backups ran, not tested for recoverability |
Each metric sounds valuable. Each hides what actually matters.
The Uptime Deception
Reporting “99.9% server uptime” creates false confidence when application-layer availability is only 95%. The server ran. Users couldn’t work.
| Measurement | What It Captures | What It Misses |
|---|---|---|
| Server uptime | Hardware/OS running | Application availability |
| Network uptime | Connectivity exists | Adequate performance |
| Service uptime | Process running | Functionality working |
| User experience | Perceived availability | Root cause visibility |
True availability measurement requires end-to-end perspective: can users do what they need to do? Infrastructure metrics are components of that answer, not the answer itself.
The 30-Day Lag Problem
Reports often lag reality by 30 days. Monthly reporting cadence means issues that emerged on the 1st aren’t visible until the following month’s report.
| Lag Duration | Risk Level | Appropriate For |
|---|---|---|
| Real-time | Lowest | Active monitoring, critical systems |
| Daily | Low | Operational awareness |
| Weekly | Medium | Trend identification |
| Monthly | High | Strategic review only |
| Quarterly | Very High | Historical analysis only |
Monthly reports serve historical review. They’re useless for active risk management. The incident that will hurt you next week doesn’t appear until the report after it happened.
The Measurement-Reality Gap
What gets measured gets managed. What doesn’t get measured doesn’t get managed.
Commonly measured:
- Response time (easy to capture from ticketing system)
- Resolution time (same)
- Ticket volume (same)
- Uptime percentage (monitoring system provides)
- Backup completion (backup tool reports)
Rarely measured:
- User satisfaction with resolution quality
- Accuracy of first response
- Recurrence of resolved issues
- Time to actually restored productivity
- Preventive work effectiveness
- Technical debt accumulation
Gap means organizations optimize for measurable metrics while unmeasured dimensions deteriorate.
The Cherry-Picking Pattern
Report creators select what to include. The selection can bias toward favorable narratives:
Selective timeframes. Report the quarter that looked good.
Metric selection. Highlight metrics that succeeded, omit those that didn’t.
Baseline manipulation. Compare to periods that make current look better.
Definition flexibility. “Resolution” means ticket closed, not problem solved.
Exception exclusion. Major incidents become footnotes rather than focus.
Detecting cherry-picking requires consistency over time. When report format or metrics change frequently, ask why.
The Context Deficit
Numbers without context mislead.
“We closed 500 tickets this month.”
Context needed: How many opened? What was the backlog change? How complex were they?
“Average resolution time improved by 20%.”
Context needed: Did ticket complexity decrease? Did ticket volume decrease?
“No security incidents this month.”
Context needed: How many attacks were attempted? What detection capability exists?
“99.9% backup success rate.”
Context needed: How many backup jobs ran? What constitutes success? When was recovery tested?
Reports that provide numbers without context invite favorable interpretation.
Building Meaningful Metrics
Metrics that inform decisions share characteristics:
Outcome-oriented. Measure what matters to business, not what’s easy to count.
Trend-visible. Show direction over time, not snapshots.
Actionable. Point toward specific improvements if unfavorable.
Verified. Subject to validation, not just MSP assertion.
Contextualized. Include baseline, comparison, and explanation.
| Business Question | Vanity Metric | Meaningful Metric |
|---|---|---|
| Are systems reliable? | Server uptime % | User-reported productivity hours lost |
| Are problems solved? | Tickets closed | Ticket reopen rate |
| Are we secure? | Zero breaches reported | Mean time to detect test scenarios |
| Are backups working? | Backup completion % | Successful restore test results |
| Is service improving? | Response time average | User satisfaction trend |
The QBR Reality Check
Quarterly Business Reviews should provide honest assessment. Often, they provide curated performance theater.
Signs of QBR theater:
Only green metrics shown. Unfavorable metrics hidden.
Historical rewriting. Past commitments quietly revised.
Forward deflection. Problems acknowledged as “being addressed.”
Comparison avoidance. No benchmarking against standards or competition.
Question discouragement. Deep questions receive shallow answers.
The QBR serves governance. Governance requires honest assessment. Theater defeats the purpose.
The Independent Verification Option
Some organizations implement independent verification of MSP-reported metrics:
User satisfaction surveys. Direct feedback bypassing MSP interpretation.
Independent monitoring. Third-party tools validating MSP claims.
Audit exercises. Periodic deep-dive verification of reported data.
Spot checking. Random validation of specific tickets or incidents.
External benchmarking. Comparison against industry standards.
Verification costs money. The cost may be worthwhile if current reporting inspires doubt.
Negotiating Better Reporting
Contract provisions that improve reporting:
Report customization rights. Specify metrics you want, not just accept defaults.
Raw data access. Access to underlying data, not just summaries.
Real-time dashboard. Live visibility, not monthly snapshots.
Trend requirements. Multi-period comparison mandatory.
Audit rights. Ability to verify reported data.
Penalty for inaccuracy. Consequences for materially incorrect reporting.
The negotiation happens at contracting, not after problems emerge. The MSP that resists reporting transparency may have reasons.
The Confidence Calibration
How confident should you be in MSP reports?
High confidence appropriate when:
- Multiple independent data sources confirm
- Trend consistent over extended periods
- Methodology is documented and stable
- Spot checks consistently validate
- User experience aligns with reported metrics
Low confidence warranted when:
- Single source, no verification
- Frequent methodology changes
- Reports conflict with user feedback
- Spot checks find discrepancies
- Questions about reports receive defensive responses
Calibrate confidence to evidence, not to desire for good news.
Sources
- Executive trust in IT reports: Forrester research
- Vanity metrics patterns: IT reporting analysis
- Reporting lag impacts: Operational monitoring studies