You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Header-based content negotiation is not sufficient. Modern LLM crawlers, such as Grok, frequently spoof standard browser headers to bypass detection measures. This neglects our reliance on User-Agent or Accept string detection to serve pure Markdown (as referenced in [WEB-4447] Add MDX to Markdown transpilation with content negotiation #3000). Consequently, we must pivot from gating content to ensuring Universal Content Accessibility.
Claude code uses both websearch and webfetch tools to obtain up-to-date information. When searching for docs on demand, it doesn't look for llms.txt instead uses websearch tool that depends on third-party providers, such as Google Search, to generate relevant links for a given query.
In conclusion, supporting only .md content isn’t sufficient — there should also be a fallback to HTML documentation that returns relevant content. This will be used by emerging LLMs and ensures proper indexing by search engines.
Phase 2 (Optimization): Tree-shake and code-split - 25% reduction
Phase 3 (Advanced): Critical CSS only - 10% reduction
🚀 Phase 1: Extract CSS Externally (1-2 days)
Goal
Move inline CSS to external file, load it asynchronously after content
Impact
Page size: 437KB → 180KB (-59%)
Initial HTML: Content-first structure
Crawlers: See documentation immediately
Step 1.1: Configure CSS Extraction
File: gatsby-config.ts
Action: Add CSS extraction plugin
// gatsby-config.tsimporttype{GatsbyConfig}from'gatsby';constconfig: GatsbyConfig={plugins: [// ADD THIS PLUGIN (insert after 'gatsby-plugin-postcss'){resolve: 'gatsby-plugin-extract-css',options: {// Extract CSS to external file instead of inlining// This will create a /styles.css fileignoreOrder: true,// Ignore CSS order warnings},},// ... rest of existing plugins'gatsby-plugin-postcss','gatsby-plugin-image',// etc.],};exportdefaultconfig;
Install dependency:
npm install gatsby-plugin-extract-css --save-dev
# or
yarn add gatsby-plugin-extract-css --dev
Result: All CSS extracted to /styles.{hash}.css, linked via <link> tag instead of inlined.
Step 1.2: Modify Layout to Load CSS Async
File: src/components/Layout/Layout.tsx
Current (imports CSS, gets inlined):
import'../../styles/global.css';// ← This gets inlined
Change to (load via Helmet after content):
importReactfrom'react';import{Helmet}from'react-helmet';constLayout=({ children })=>{return(<><Helmet>{/* Preload CSS file (starts download early, doesn't block render) */}<linkrel="preload"href="/styles.css"as="style"onLoad="this.onload=null;this.rel='stylesheet'"/>{/* Fallback for no-JS */}<noscript>{`<link rel="stylesheet" href="/styles.css">`}</noscript></Helmet>{/* Content renders FIRST, before CSS */}{children}</>);};exportdefaultLayout;
Alternative (even better - async load with loadCSS):
importReact,{useEffect}from'react';import{Helmet}from'react-helmet';constLayout=({ children })=>{useEffect(()=>{// Load CSS asynchronously after page loadsconstlink=document.createElement('link');link.rel='stylesheet';link.href='/styles.css';document.head.appendChild(link);},[]);return(<><Helmet>{/* Critical inline CSS only (see Phase 3) */}<style>{` /* Minimal critical CSS here - ~5KB */ body { font-family: sans-serif; margin: 0; } .container { max-width: 1200px; margin: 0 auto; } `}</style></Helmet>{children}</>);};exportdefaultLayout;
Step 1.3: Update HTML Structure (Content First)
File: src/html.js (create if doesn't exist)
Gatsby allows customizing the base HTML template:
// src/html.jsimportReactfrom'react';importPropTypesfrom'prop-types';exportdefaultfunctionHTML(props){return(<html{...props.htmlAttributes}><head><metacharSet="utf-8"/><metahttpEquiv="x-ua-compatible"content="ie=edge"/><metaname="viewport"content="width=device-width, initial-scale=1, shrink-to-fit=no"/>{/* Meta tags, title - all the SEO stuff */}{props.headComponents}{/* CRITICAL CSS ONLY - inline just what's needed */}<styledangerouslySetInnerHTML={{__html: ` /* Minimal above-the-fold CSS */ body { margin: 0; font-family: system-ui, sans-serif; } .docs-content { max-width: 800px; margin: 0 auto; padding: 20px; } h1 { font-size: 2rem; font-weight: 700; } code { background: #f5f5f5; padding: 2px 6px; border-radius: 3px; } `}}/>{/* CSS loaded async after content */}<linkrel="preload"href="/styles.css"as="style"onLoad="this.rel='stylesheet'"/><noscript><linkrel="stylesheet"href="/styles.css"/></noscript></head><body{...props.bodyAttributes}>{/* CONTENT COMES FIRST - this is what crawlers see immediately */}{props.preBodyComponents}<divkey="body"id="___gatsby"dangerouslySetInnerHTML={{__html: props.body}}/>{/* Scripts at the end */}{props.postBodyComponents}</body></html>);}HTML.propTypes={htmlAttributes: PropTypes.object,headComponents: PropTypes.array,bodyAttributes: PropTypes.object,preBodyComponents: PropTypes.array,body: PropTypes.string,postBodyComponents: PropTypes.array,};
Step 1.4: Test the Changes
# Clean cache
gatsby clean
# Build for production
gatsby build
# Serve and test
gatsby serve
# Check page size
curl -s http://localhost:9000/docs/chat/getting-started/android | wc -c
# Expected: ~150-180KB (down from 437KB)
// tailwind.config.jsconstablyUIConfig=require('@ably/ui/tailwind.config.js');module.exports={presets: [ablyUIConfig],content: ['./src/pages/**/*.{js,jsx,ts,tsx,mdx}','./src/components/**/*.{js,jsx,ts,tsx}','./src/templates/**/*.{js,jsx,ts,tsx}','./data/**/*.{js,ts}','./content/**/*.mdx',// If you have MDX content// Include @ably/ui components'./node_modules/@ably/ui/**/*.{js,jsx,ts,tsx}',],// IMPORTANT: Enable JIT mode for faster builds and smaller outputmode: 'jit',// Remove unused variantssafelist: [// Only safelist classes that are dynamically generated// Example: 'bg-blue-500', 'text-red-600'],theme: {extend: {// Your custom theme},},};
Add to gatsby-config.ts (ensure Tailwind processes correctly):
importReact,{useEffect,useState}from'react';constCode=({ children, language })=>{const[syntaxLoaded,setSyntaxLoaded]=useState(false);useEffect(()=>{// Dynamically import syntax highlighting CSS only when neededif(!syntaxLoaded){import('@ably/ui/core/utils/syntax-highlighter.css').then(()=>setSyntaxLoaded(true));}},[]);return(<preclassName={`language-${language}`}><code>{children}</code></pre>);};exportdefaultCode;
Better approach - Use dynamic import for Prism.js itself:
importReact,{useEffect,useState}from'react';constCode=({ children, language ='javascript'})=>{const[highlighted,setHighlighted]=useState(false);useEffect(()=>{// Only load Prism.js for this languagePromise.all([import('prismjs'),import(`prismjs/components/prism-${language}`),import('prismjs/themes/prism-tomorrow.css'),// Just one theme]).then(([Prism])=>{Prism.highlightAll();setHighlighted(true);});},[language]);return(<preclassName={`language-${language}`}><code>{children}</code></pre>);};exportdefaultCode;
/* src/styles/global.css - MINIMAL VERSION *//* Tailwind base - necessary */@import'tailwindcss/base';
@import'tailwindcss/components';
@import'tailwindcss/utilities';
/* Ably UI reset only */@import'@ably/ui/reset/styles.css';
/* DO NOT import full @ably/ui/core/styles.css *//* Instead, import per-component in component files */
Configure webpack to code-split (Gatsby 5 does this automatically for component imports)
Expected reduction: @ably/ui: 150KB → 40KB (only used components)
Step 2.4: Optimize Component CSS
File: Component CSS files (26 files, ~104KB)
Strategy: Use CSS Modules + tree-shaking
Example - Before (src/components/Menu/styles.css):
/* Lots of unused styles for all menu variants */
.menu { }
.menu-item { }
.menu-item-active { }
.menu-item-hover { }
/* ... 500 more lines */
After - Use CSS Modules properly:
/* src/components/Menu/Menu.module.css *//* Only styles actually used by Menu component */
.menu {
composes: flex flex-col from global; /* Use Tailwind */
}
.menuItem {
/* Only custom styles Tailwind doesn't provide */transition: all 0.2s;
}
Configure Gatsby to use CSS Modules:
// gatsby-config.ts (this should already work by default){resolve: 'gatsby-plugin-postcss',options: {cssLoaderOptions: {modules: {auto: true,// Enable CSS Modules for *.module.css fileslocalIdentName: '[local]_[hash:base64:5]',},},},},
<!-- Preconnect to font CDN --><linkrel="preconnect" href="https://fonts.googleapis.com" /><linkrel="preconnect" href="https://fonts.gstatic.com" crossOrigin="anonymous" /><!-- Preload font files --><linkrel="preload"
as="style"
href="https://fonts.googleapis.com/css2?family=Manrope:wght@400;600;700&display=swap"
/><!-- Load async --><linkrel="stylesheet"
href="https://fonts.googleapis.com/css2?family=Manrope:wght@400;600;700&display=swap"
media="print"
onLoad="this.media='all'"
/><!-- Fallback system font inline --><style>body {
font-family: -apple-system, BlinkMacSystemFont,'Segoe UI', Roboto, sans-serif;
}
</style>
Or self-host fonts (even better - no external requests):
# Download fonts to /static/fonts/# Add to CSS:
@font-face {
font-family: 'Manrope';
src: url('/fonts/manrope-regular.woff2') format('woff2');
font-weight: 400;
font-display: swap;
}
Step 3.3: Defer Non-Critical Resources
Images - use lazy loading:
import{GatsbyImage}from'gatsby-plugin-image';// Already optimized with gatsby-plugin-image// Ensure loading="lazy" is set<GatsbyImageimage={imageData}alt="..."loading="lazy"/>
Initial HTML: 35KB (-92% from original 437KB!)
├─ Critical CSS: 5KB (14%)
├─ Content: 30KB (86%)
Deferred Resources (cached):
├─ Full CSS: 80KB
├─ Fonts: 40KB
├─ JavaScript: 150KB
First Contentful Paint: <0.5s (was 2-3s) Crawler Experience: Content in first 35KB
📋 Implementation Checklist
Phase 1 (1-2 days) ✅
Install gatsby-plugin-extract-css
Configure plugin in gatsby-config.ts
Remove CSS import from Layout.tsx
Add async CSS loading via Helmet
Create custom src/html.js with content-first structure
Test build and verify page size reduction
Verify content visible before full CSS loads
Phase 2 (3-5 days) 🔧
Configure Tailwind JIT mode
Add PostCSS PurgeCSS plugin
Update Tailwind content paths
Move syntax highlighting to dynamic imports
Refactor @ably/ui imports (per-component)
Update global.css (minimal imports only)
Optimize component CSS files
Test all pages still render correctly
Measure CSS file size reduction
Phase 3 (2-3 days) 🎯
Install gatsby-plugin-critical
Configure critical CSS extraction
Optimize font loading (preload/async)
Self-host fonts (optional)
Add resource hints (preconnect, dns-prefetch)
Defer non-critical JavaScript
Test perceived performance
Validate with Lighthouse
🧪 Testing & Validation
Automated Tests
1. Page Size Check:
#!/bin/bash# test-page-size.sh
URL="http://localhost:9000/docs/chat/getting-started/android"
SIZE=$(curl -s "$URL"| wc -c)echo"Page size: $SIZE bytes"if [ $SIZE-lt 100000 ];thenecho"✅ PASS: Page size under 100KB"elseecho"❌ FAIL: Page size over 100KB"exit 1
fi
2. Content-First Validation:
#!/bin/bash# test-content-first.sh
URL="http://localhost:9000/docs/chat/getting-started/android"
FIRST_100KB=$(curl -s "$URL"| head -c 100000)# Check if documentation content appears in first 100KBifecho"$FIRST_100KB"| grep -q "implementation.*chat";thenecho"✅ PASS: Content found in first 100KB"elseecho"❌ FAIL: Content not in first 100KB"exit 1
fi
3. CSS Extraction Check:
#!/bin/bash# test-css-extracted.sh
URL="http://localhost:9000/docs/chat/getting-started/android"
HTML=$(curl -s "$URL")# Check for external CSS linkifecho"$HTML"| grep -q '<link.*rel="stylesheet".*href=.*\.css';thenecho"✅ PASS: CSS extracted to external file"elseecho"❌ FAIL: CSS still inlined"exit 1
fi# Check inline CSS size is small
INLINE_CSS_SIZE=$(echo "$HTML"| grep -oP '<style[^>]*>.*?</style>'| wc -c)if [ $INLINE_CSS_SIZE-lt 10000 ];thenecho"✅ PASS: Inline CSS under 10KB"elseecho"❌ FAIL: Too much inline CSS ($INLINE_CSS_SIZE bytes)"exit 1
fi
Manual Testing
1. Lighthouse Audit:
# Install Lighthouse CLI
npm install -g lighthouse
# Run audit
lighthouse http://localhost:9000/docs/chat/getting-started/android \
--output html \
--output-path ./lighthouse-report.html
# Target scores:# - Performance: >90# - First Contentful Paint: <1.5s# - Largest Contentful Paint: <2.5s
2. Crawler Simulation:
# Simulate crawler (no JavaScript, no CSS)
curl -s http://localhost:9000/docs/chat/getting-started/android | \
w3m -dump -T text/html | \
head -50
# Should see:# - Page title# - Documentation content# - Code examples# - NOT: CSS, JSON data, JavaScript
webfetchcomplains that the response contains only CSS. See related issue: https://ably.atlassian.net/browse/FTF-227User-AgentorAcceptstring detection to serve pure Markdown (as referenced in [WEB-4447] Add MDX to Markdown transpilation with content negotiation #3000). Consequently, we must pivot from gating content to ensuring Universal Content Accessibility.websearchandwebfetchtools to obtain up-to-date information. When searching for docs on demand, it doesn't look forllms.txtinstead useswebsearchtool that depends on third-party providers, such asGoogle Search, to generate relevant links for a given query..mdcontent isn’t sufficient — there should also be a fallback toHTML documentationthat returns relevant content. This will be used by emerging LLMs and ensures proper indexing by search engines.Problem: Pages are 437KB with 91% CSS overhead
Impact:
Comprehensive Guide to Reduce Page Bloat from 437KB to <50KB
🎯 Solution Overview
Target Metrics
Three-Phase Approach
🚀 Phase 1: Extract CSS Externally (1-2 days)
Goal
Move inline CSS to external file, load it asynchronously after content
Impact
Step 1.1: Configure CSS Extraction
File:
gatsby-config.tsAction: Add CSS extraction plugin
Install dependency:
npm install gatsby-plugin-extract-css --save-dev # or yarn add gatsby-plugin-extract-css --devResult: All CSS extracted to
/styles.{hash}.css, linked via<link>tag instead of inlined.Step 1.2: Modify Layout to Load CSS Async
File:
src/components/Layout/Layout.tsxCurrent (imports CSS, gets inlined):
Change to (load via Helmet after content):
Alternative (even better - async load with loadCSS):
Step 1.3: Update HTML Structure (Content First)
File:
src/html.js(create if doesn't exist)Gatsby allows customizing the base HTML template:
Step 1.4: Test the Changes
Verify in browser DevTools:
Phase 1 Results
Before:
After Phase 1:
Impact:
🔧 Phase 2: Tree-Shake & Optimize (3-5 days)
Goal
Reduce CSS file size by removing unused styles
Impact
Step 2.1: Optimize Tailwind CSS
File:
tailwind.config.jsCurrent issue: Generating all Tailwind utilities
Solution: Configure purge properly
Add to gatsby-config.ts (ensure Tailwind processes correctly):
Install PurgeCSS:
Expected reduction: Tailwind CSS: 150KB → 30KB
Step 2.2: Lazy Load Syntax Highlighting
File:
src/styles/global.cssCurrent (loads ALL syntax highlighting):
Solution: Load dynamically only when code blocks exist
Remove from global.css, add to code component:
File:
src/components/blocks/software/Code/Code.tsxBetter approach - Use dynamic import for Prism.js itself:
Expected reduction: Syntax highlighting: 50KB → 5KB (per language, loaded on-demand)
Step 2.3: Code-Split @ably/ui Components
File:
src/styles/global.cssCurrent (imports entire @ably/ui library):
Solution: Import only what each page needs
Create a minimal global.css:
Import component styles where used:
File:
src/components/CookieConsent.tsxFile:
src/components/Slider.tsxConfigure webpack to code-split (Gatsby 5 does this automatically for component imports)
Expected reduction: @ably/ui: 150KB → 40KB (only used components)
Step 2.4: Optimize Component CSS
File: Component CSS files (26 files, ~104KB)
Strategy: Use CSS Modules + tree-shaking
Example - Before (
src/components/Menu/styles.css):After - Use CSS Modules properly:
Configure Gatsby to use CSS Modules:
Expected reduction: Component CSS: 104KB → 30KB
Phase 2 Results
After Phase 2:
Total page weight: 130KB (loaded: 50KB HTML + 80KB CSS cached)
🎯 Phase 3: Critical CSS Only (2-3 days)
Goal
Inline ONLY above-the-fold CSS, defer everything else
Impact
Step 3.1: Extract Critical CSS
Use a tool to automatically extract above-the-fold CSS
Install Critical CSS plugin:
Add to gatsby-config.ts:
How it works:
Step 3.2: Optimize Font Loading
File:
src/html.jsor via HelmetCurrent (fonts may block render):
Optimized (preload + async):
Or self-host fonts (even better - no external requests):
Step 3.3: Defer Non-Critical Resources
Images - use lazy loading:
Third-party scripts - load async:
Phase 3 Results
Final Result:
First Contentful Paint: <0.5s (was 2-3s)
Crawler Experience: Content in first 35KB
📋 Implementation Checklist
Phase 1 (1-2 days) ✅
gatsby-plugin-extract-cssgatsby-config.tsLayout.tsxsrc/html.jswith content-first structurePhase 2 (3-5 days) 🔧
Phase 3 (2-3 days) 🎯
gatsby-plugin-critical🧪 Testing & Validation
Automated Tests
1. Page Size Check:
2. Content-First Validation:
3. CSS Extraction Check:
Manual Testing
1. Lighthouse Audit:
2. Crawler Simulation:
3. WebPageTest:
📊 Expected Results Summary
🎓 Best Practices Going Forward
1. CSS Strategy
2. Build Process
3. Content Priority
4. Monitoring
🔗 Additional Resources
Gatsby Documentation
Tailwind CSS
Performance
Tools
💬 Support & Questions
For questions or issues during implementation: