react

Twelve Tabs, One Feed, and a Bad Cron Job

Pandiyan MuruganLeave a comment

How I fixed the way WealthWire fetches news, and why it took me two wrong attempts to get there

A few years ago I built WealthWire for myself. I follow a lot of Indian financial news, SEBI filings, market commentary, budget coverage, and I was tired of having twelve tabs open. So I made a small aggregator that pulls from the RSS feeds I care about and shows them in one place.

It has stayed a side project. No team, no funding, just me and a handful of RSS feeds. But even at that scale, the way you fetch data matters, and I got it wrong twice before landing on something that actually holds up.

Mistake one: fetching per user

The first version did what felt natural. You open the site, the server fetches the feeds you follow, normalizes them, and renders the page. Simple.

It also has a shape that falls apart under any load:

feed fetches = active users × feeds per user × page loads

Two people reading the same Economic Times RSS feed at the same moment triggered two identical fetches. Same URL, same response, fetched twice. Ten users, ten fetches. The feed was returning byte-identical XML every time and I was hitting it once per person asking.

The fix is obvious once you see it: the data isn’t per-user, so the fetching shouldn’t be either.

Mistake two: cron

The standard answer is a scheduled job. Run it every 30 minutes, fetch every feed once, write the results to a database. Users read from there instead of from the feeds directly. The store is never more than 30 minutes stale, and you’ve broken the link between user count and fetch count.

This was genuinely better. But a scheduler has one blind spot: it doesn’t know if anyone is actually reading.

WealthWire is a personal project with uneven traffic. Indian markets are open from 9:15 a.m. to 3:30 p.m. on weekdays. Outside those hours, and especially late at night or on weekends, there is almost nobody on the site. The cron job didn’t know that. It fired at 2 a.m., at 3 a.m., through every public holiday, through every weekend. Fetching news that nobody was about to read, refreshing a cache nobody was about to hit.

It worked. It was also paying a quiet, permanent tax for freshness that had no readers.

What I do now: refresh on read

The current approach keeps the best part of cron (one fetch serves everyone) but ties it to actual demand instead of a clock.

The data refreshes when someone asks for it, and only if the last fetch is old enough to need refreshing.

There is no scheduler running in the background. Every incoming request checks one thing first: is the cached copy still fresh? If yes, it gets served immediately with no further work. If no, one request fetches the feeds and writes a new copy. Everyone else in that same moment gets the slightly older copy, and a few seconds later the fresh one is there for them too.

How the layers fit together

A single request passes through three caches before anything actually hits an RSS feed. Each one is cheaper to serve from than the one after it:

Browser cache holds the last response in localStorage with a 30-minute TTL. A user refreshing the page within that window never even leaves their own browser.

CDN / edge cache sits between the browser and the server. With stale-while-revalidate and a matching s-maxage, the CDN can serve a cached response to most visitors while revalidating in the background. No request reaches the server at all.

Server cache (a durable key-value store or blob storage) is the last line before an actual feed fetch. The server checks a timestamp: if the cached data is under 30 minutes old, it returns it. If not, it fetches, writes a fresh copy, and returns that.

A request only reaches a real RSS feed URL if it misses every layer in sequence. On a site with any regular traffic, that is a small fraction of requests.

What happens inside a cache miss

When the server cache is stale, three things happen in order:

Freshness check fails. The timestamp says the data is old. The request needs to fetch.

Lock acquisition. Before fetching, the request tries to claim a short-lived lock (around 60 seconds). Only one request can hold it at a time.

Lock won: fetch and write. The request that wins the lock pulls each feed, parses and normalizes the XML into a common format, and writes the result back to the cache with a current timestamp. If one feed fails (a 404, a timeout, malformed XML), that feed is skipped. The others still write. A broken source costs you that source, not the whole page.

Lock lost: serve what’s there. Every other concurrent request that checked the timestamp at the same moment doesn’t wait. It reads whatever is already in the cache, which is at most 30 minutes old, and returns immediately. By the time the next request comes in, the fresh copy is already there.

Why the lock matters more than anything else

Leave the lock out and the pattern breaks at the worst time.

Imagine 200 people opening the site the moment NSE opens for the day. The cache is 31 minutes old. Without a lock, all 200 requests see stale data, all 200 decide to fetch, and some publisher’s RSS endpoint gets 200 simultaneous requests for the same XML file. You have gone from “one fetch per 30 minutes” to “200 fetches in the same second.” The pattern has made things worse than cron, precisely at the moment your traffic is highest.

The lock collapses that to one fetch. The 199 other requests read the cache that was already there, which is 31 minutes old rather than 30. Nobody notices. The publisher’s server sees one request. A few seconds later, fresh data is in the cache for everyone.

The trade-off worth naming

Freshness is now tied to traffic. If nobody visits WealthWire for six hours, the feeds don’t get pulled for six hours. The next visitor will see data that is a bit older than the stated 30-minute TTL, and will wait a moment while a fresh fetch completes.

For a personal project tracking market hours, that is exactly the right trade. There is no one to serve stale data to at 3 a.m., so there is no cost to not refreshing. The cache goes cold when readers do, and warms back up the moment someone shows up.

If freshness genuinely cannot depend on someone visiting (financial compliance, anything with a regulatory clock), this pattern alone isn’t enough. You would want to keep a lightweight scheduled “warm the cache” ping for off-hours, and use this pattern to handle everything during live traffic.

What to watch once it’s running

A few metrics worth tracking to know the pattern is actually working:

Cache hit ratio per layer. Measure browser, CDN, and server hits separately. If everything is hitting the server cache, the CDN config probably needs work. If everything is hitting the CDN, great.

Outbound feed requests per day. This is the number the whole thing exists to shrink. With a 30-minute TTL the theoretical ceiling is 48 batches per day. If you’re near that number, something is missing at the layers in front.

Lock contention rate. How often does a request find the lock already held? Some contention is fine and expected. A lot of it means traffic is consistently bunching at TTL expiry, and you may want a shorter lock timeout or staggered TTLs per feed.

P95 latency on misses. The miss path (feed fetch plus parse plus write) has real latency. It should be bounded by the slowest feed’s timeout. If it keeps growing, a slow publisher is bleeding into the user experience.

Per-feed failure rate. RSS feeds break quietly. A feed that stops responding just looks like no new articles, not an error. Log fetch outcomes per feed or you will lose visibility into what is actually fresh.

~ tl;tr;

Cron solved the right problem (one fetch, many users) but kept doing work when there was no reason to. The fix isn’t a fancier scheduler. It’s removing the scheduler entirely and letting requests decide when freshness is actually needed.

No background workers. No queue. Just a timestamp, a lock, and a cache in the request path. The site does less work when nobody is reading, which is most of the time.

Built by one person. Runs on quiet infrastructure. Serves Indian finance news to anyone who wants it in one place: wealthwire.in

Fixing the “Maximum Update Depth Exceeded” Error in React

Pandiyan MuruganLeave a comment

How to debug and resolve infinite re-render loops in React components

The Problem: When React Components Go Haywire 🔥

Picture this: You’re working on a React application, everything seems to be working fine, and then suddenly your browser console explodes with this dreaded error:

Warning: Maximum update depth exceeded. This can happen when a component calls setState inside useEffect, but useEffect either doesn't have a dependency array, or one of the dependencies changes on every render.

This error indicates that your component is stuck in an infinite re-render loop, making your application unusable and creating a frustrating user experience.

Understanding the Root Cause 🕵️‍♂️

This error commonly occurs in components that manage dynamic lists or collections, such as sidebar navigation, data tables, or filtered content. Let’s examine a typical scenario where this happens:

The Problematic Code

// ❌ BEFORE: The problematic implementation
const filteredItems = items?.filter(/* filtering logic */) || [];
const groupedItems = filteredItems.reduce(/* grouping logic */, {});
const sortedGroups = Object.entries(groupedItems).sort(/* sorting logic */);

useEffect(() => {
  if (sortedGroups.length > 0) {
    const allGroupNames = sortedGroups.map(([groupName]) => groupName);
    setExpandedGroups(new Set(allGroupNames));
  }
}, [sortedGroups]); // 🚨 This dependency changes on every render!

Why This Caused Infinite Re-renders

  1. Unstable Dependencies: The sortedGroups array was being recreated on every render because the filtering, grouping, and sorting operations weren’t memoized.
  2. State Update Trigger: Every time sortedGroups changed (which was every render), the useEffect would fire and call setExpandedGroups.
  3. Re-render Cascade: The state update would trigger another re-render, which would recreate sortedGroups, which would trigger the useEffect again, and so on…
  4. React’s Safety Net: After detecting this pattern, React threw the “Maximum update depth exceeded” error to prevent an infinite loop from crashing the browser.

The Solution: Memoization and Smart State Updates 🛠️

Here’s a multi-pronged approach to fix this issue:

1. Memoize Expensive Computations

The first step is to wrap expensive operations in useMemo to ensure they only recalculate when their actual dependencies change:

// ✅ AFTER: Memoized filtering
const filteredItems = useMemo(
  () =>
    items?.filter(
      (item: any) =>
        (item.title || '').toLowerCase().includes(searchQuery.toLowerCase()) ||
        (item.description || '')
          .toLowerCase()
          .includes(searchQuery.toLowerCase()) ||
        (item.category?.name || '')
          .toLowerCase()
          .includes(searchQuery.toLowerCase())
    ) || [],
  [items, searchQuery] // Only recalculate when items or searchQuery change
);

// ✅ AFTER: Memoized grouping
const groupedItems = useMemo(() => {
  return filteredItems.reduce((acc: any, item: any) => {
    const groupName = item.category?.name || 'Uncategorized';
    if (!acc[groupName]) {
      acc[groupName] = {
        category: item.category,
        items: [],
      };
    }
    acc[groupName].items.push(item);
    return acc;
  }, {} as Record<string, { category: any; items: any[] }>);
}, [filteredItems]);

// ✅ AFTER: Memoized sorting
const sortedGroups = useMemo(() => {
  return Object.entries(groupedItems).sort(([a], [b]) => {
    if (a === 'Uncategorized') return 1;
    if (b === 'Uncategorized') return -1;
    return a.localeCompare(b);
  }) as [string, { category: any; items: any[] }][];
}, [groupedItems]);

2. Create Stable Dependencies

Extract the group names into a separate memoized value to create a stable dependency for useEffect:

// ✅ AFTER: Stable dependency for useEffect
const groupNames = useMemo(() => {
  return sortedGroups.map(([groupName]) => groupName);
}, [sortedGroups]);

3. Implement Smart State Updates

The most crucial fix is implementing a comparison check before updating the state:

// ✅ AFTER: Smart state updates with comparison
useEffect(() => {
  if (groupNames.length > 0) {
    setExpandedGroups(prev => {
      // Only update if the group names have actually changed
      const currentNames = Array.from(prev).sort();
      const newNames = [...groupNames].sort();

      if (JSON.stringify(currentNames) !== JSON.stringify(newNames)) {
        return new Set(groupNames);
      }
      return prev; // 🎯 Return previous state if no change needed
    });
  }
}, [groupNames]);

The Key Insights 💡

1. Always Return Previous State When No Change is Needed

The most important lesson here is that when using functional state updates, always return the previous state if no actual change is needed. This prevents unnecessary re-renders.

2. Memoize Expensive Operations

Operations like filtering, sorting, and reducing large arrays should be wrapped in useMemo to prevent unnecessary recalculations.

3. Be Careful with Object and Array Dependencies

Objects and arrays are compared by reference in React. Even if their contents are the same, if they’re recreated on each render, React considers them different.

4. Use Functional State Updates for Comparisons

When you need to compare the current state with new values before updating, use the functional form of setState:

setState(prev => {
  // Compare prev with new value
  if (shouldUpdate) {
    return newValue;
  }
  return prev; // Important: return previous state
});

Performance Benefits 📈

After implementing these fixes, you can expect:

  • Eliminated infinite re-renders: Components only re-render when necessary
  • Improved performance: Memoization reduces unnecessary computations
  • Better user experience: UI interactions become smooth and responsive
  • Reduced CPU usage: No more constant re-rendering cycles

Testing the Fix 🧪

To verify the fix works properly:

  1. Monitor React DevTools: Check that components aren’t re-rendering unnecessarily
  2. Add console logs: Temporarily log when expensive operations run
  3. Test edge cases: Ensure the fix works with empty data, single items, etc.
  4. Performance profiling: Use React DevTools Profiler to measure render times

Best Practices to Prevent This Issue 🛡️

  1. Use useMemo for expensive computations that depend on props or state
  2. Use useCallback for event handlers that are passed to child components
  3. Always check if state actually needs to change before calling setState
  4. Be mindful of dependencies in useEffect – ensure they’re stable
  5. Use React DevTools to identify unnecessary re-renders during development

Conclusion

The “Maximum update depth exceeded” error might seem intimidating, but it’s actually React trying to protect your application from infinite loops. By understanding the root cause – usually unstable dependencies or unnecessary state updates – you can implement targeted fixes that not only resolve the error but also improve your application’s performance.

The key takeaways from this fix:

  • Memoize expensive operations with useMemo
  • Create stable dependencies for useEffect
  • Compare before updating state and return previous state when no change is needed
  • Always profile and test your fixes to ensure they work as expected

Remember: React’s error messages are your friend. They’re designed to help you write better, more performant code. When you encounter them, take the time to understand the root cause rather than just patching the symptoms.

Have you encountered similar infinite re-render issues in your React applications? Share your experiences and solutions in the comments below!

Example Implementation

Here’s a complete example of how to implement this fix in your own components:

import { useMemo, useEffect, useState } from 'react';

function ListComponent({ items, searchQuery }) {
  const [expandedGroups, setExpandedGroups] = useState(new Set());

  // Memoized filtering
  const filteredItems = useMemo(
    () =>
      items?.filter(item =>
        item.title.toLowerCase().includes(searchQuery.toLowerCase())
      ) || [],
    [items, searchQuery]
  );

  // Memoized grouping
  const groupedItems = useMemo(() => {
    return filteredItems.reduce((acc, item) => {
      const group = item.category || 'Other';
      if (!acc[group]) acc[group] = [];
      acc[group].push(item);
      return acc;
    }, {});
  }, [filteredItems]);

  // Stable dependency
  const groupNames = useMemo(() => Object.keys(groupedItems), [groupedItems]);

  // Smart state updates
  useEffect(() => {
    if (groupNames.length > 0) {
      setExpandedGroups(prev => {
        const current = Array.from(prev).sort();
        const newNames = [...groupNames].sort();

        if (JSON.stringify(current) !== JSON.stringify(newNames)) {
          return new Set(groupNames);
        }
        return prev;
      });
    }
  }, [groupNames]);

  // Rest of component...
}

Happy debugging!