​

Testing with Chatlab

Chatlab is your private testing environment where you can chat with your agent, test flows, and debug issues before customers ever see them. It’s essential for catching problems early and ensuring quality conversations.

​

What is Chatlab?

Chatlab is a sandboxed chat interface built directly into ChatCrafterAI. It replicates your live agent experience exactly—same AI, same knowledge base, same flows—but nothing goes live until you approve it. Why use Chatlab?

• Catch errors early - Find issues before deployment
• Test AI responses - Verify answer quality and accuracy
• Validate flows - Ensure buttons, forms, and logic work correctly
• Check integrations - Test Stripe payments, Shopify lookups, etc. in test mode
• Verify personality - Make sure your agent sounds right
• Test edge cases - See how agent handles unusual inputs
• No customer impact - All testing is private and won’t affect real users

​

Access Chatlab

Opening Chatlab takes 30 seconds:

1. Navigate to your agent on the dashboard
2. Click “Chatlab” button in the top-right corner
3. Start typing your test messages in the chat window
4. Watch responses appear with full debugging information
5. Review logs to understand what’s happening

The interface is identical to your live web widget—what you see is exactly what customers will experience.

​

What You Can Test

​

Knowledge Base Testing

Your agent should accurately answer questions from your uploaded documents. Here’s what to test:

• Direct questions - “What is your refund policy?” → Should return accurate answer
• Variations - Ask the same question differently → Same answer
• Partial knowledge - Ask about related topics → Should acknowledge limitations
• Out of scope - Ask something not in your KB → Should politely say it doesn’t know
• Multi-document answers - Does it combine info from multiple sources? → Test compound questions

Example: If your KB has product details and pricing, ask “What’s the most expensive widget and what’s its warranty?” to verify it can piece together information.

​

Flow & Button Testing

If you’re using Rich-Only or Hybrid mode with predefined flows:

• Button clicks - Do all buttons work and lead to correct paths?
• Form submission - Can users fill forms and submit data?
• Conditional logic - Does flow branch correctly based on user input?
• Error handling - What happens if user submits incomplete form?
• Flow loops - Can user ask clarifying questions within a flow?
• Navigation - Can user go back or exit a flow?

Test every button path manually to ensure no dead ends.

​

Integration Testing

Test real-world integrations in safe test mode: Stripe Payments

• Try checkout flow with test card numbers
• Verify amount is correct
• Check invoice creation
• Test card decline scenarios

Shopify Product Lookup

• Ask for specific products by name
• Verify product details are accurate
• Check if inventory is current
• Test with products that don’t exist

Zendesk Escalation

• Trigger ticket creation
• Verify ticket appears in Zendesk
• Check ticket details are captured correctly
• Test custom fields and metadata

​

Personality & Tone

Verify your agent’s personality aligns with your brand:

• Professionalism level - Is it too casual? Too formal?
• Response length - Are answers concise or rambling?
• Helpfulness - Does it proactively offer next steps?
• Error messages - When agent can’t help, is it graceful?
• Humor/warmth - Does personality feel authentic?

Read responses aloud to hear how they sound.

​

Edge Cases & Error Scenarios

Test unusual situations:

• Very long questions (100+ words) - Does AI still understand?
• Nonsense input (“aslkdjfl hkj”) - Graceful error handling?
• Repeated requests - Can user ask same thing multiple times?
• Rapid clicking - What if user clicks buttons 10 times fast?
• Special characters - How does agent handle emojis, symbols?
• Different languages - If multilingual, test each language

​

Debug Mode - Understanding What’s Happening

Chatlab’s debug panel shows exactly how your agent processes each message:

​

Debug Panel Shows:

1. AI Prompt Sent

• Exact system prompt your agent received
• User message sent to AI
• Context and parameters used

2. Knowledge Base Retrieved

• Which documents were searched
• Relevance score for each match
• How many chunks were retrieved
• Actual text sent to AI as context

3. Flow Matching

• Which flow pattern matched (if any)
• Confidence score of the match
• What variables were captured

4. Integration Responses

• API calls made to Stripe, Shopify, etc.
• Response codes (200 = success, 4xx = error)
• Data returned from integrations

5. Error Messages

• Why a knowledge base search failed
• Why a flow didn’t trigger
• Why an integration call failed
• Specific error codes with explanations

How to use debug info: If a response is wrong, check the debug panel to see what context the AI received. Often you’ll find the knowledge base retrieved the wrong document—that’s a KB optimization issue, not an AI issue.

​

Testing Workflow

Follow this systematic approach:

​

Step 1: Knowledge Base Validation (15 minutes)

• Ask 10 common questions your customers will ask
• Verify each answer is accurate
• Check response quality and relevance
• Note any missing information

​

Step 2: Flow Testing (10 minutes)

• Click through each button path
• Fill out every form
• Test all conditional logic
• Verify form data is captured

​

Step 3: Integration Testing (10 minutes)

• Process a test Stripe payment
• Look up a test Shopify product
• Create a test Zendesk ticket
• Verify data appears correctly

​

Step 4: Personality Check (5 minutes)

• Read 5 responses aloud
• Do they sound like your brand?
• Are they helpful and friendly?
• Any grammar or tone issues?

​

Step 5: Edge Case Testing (10 minutes)

• Send weird input
• Rapid-click buttons
• Try long questions
• Test multilingual if applicable

​

Tips for Effective Testing

Document issues as you find them

• Take screenshots of problem areas
• Note exact questions that break the agent
• Record steps to reproduce each issue

Test like your customers

• Don’t just test “happy paths”
• Ask confusing questions
• Try things the “wrong way”
• Think like someone frustrated

Test on mobile too

• Open widget on phone/tablet
• Verify responsive layout works
• Check button sizes are clickable
• Test on different browsers

Ask a colleague to test

• Fresh eyes catch things you miss
• Non-experts expose confusing behavior
• Get honest feedback on personality

Iterate before deploying

• Each issue you find now = saved customer frustration later
• Plan to spend 30-60 minutes testing before first deploy
• Re-test after making changes

​

Common Issues Found in Testing

Problem: Agent gives irrelevant answers

• Cause: Knowledge base has too much similar content
• Fix: Improve chunk separation, add clearer headings

Problem: Forms not submitting

• Cause: Conditional logic is too strict
• Fix: Review flow requirements, simplify form fields

Problem: Integration returns error

• Cause: API keys wrong or test mode not enabled
• Fix: Verify integration setup, check API credentials

Problem: Response time is slow

• Cause: Knowledge base is too large or needs optimization
• Fix: Optimize chunks, reduce KB size, improve document quality

Problem: Agent personality doesn’t match brand

• Cause: System prompt too generic
• Fix: Refine system prompt with specific brand voice guidelines

​

Deploy When Confident

Once you’ve tested thoroughly and fixed issues:

1. Review your test results - Any remaining concerns?
2. Get approval - Have team review agent behavior?
3. Click “Deploy to Live” - Make agent live on selected channels
4. Monitor first 24 hours - Check analytics for issues
5. Keep testing post-launch - Real usage will reveal new scenarios

Remember: Thorough testing = confident deployment = happy customers.

---

Done testing? Next: Multi-Channel Deployment