Nextjs

Twelve Tabs, One Feed, and a Bad Cron Job

Pandiyan MuruganLeave a comment

How I fixed the way WealthWire fetches news, and why it took me two wrong attempts to get there

A few years ago I built WealthWire for myself. I follow a lot of Indian financial news, SEBI filings, market commentary, budget coverage, and I was tired of having twelve tabs open. So I made a small aggregator that pulls from the RSS feeds I care about and shows them in one place.

It has stayed a side project. No team, no funding, just me and a handful of RSS feeds. But even at that scale, the way you fetch data matters, and I got it wrong twice before landing on something that actually holds up.

Mistake one: fetching per user

The first version did what felt natural. You open the site, the server fetches the feeds you follow, normalizes them, and renders the page. Simple.

It also has a shape that falls apart under any load:

feed fetches = active users × feeds per user × page loads

Two people reading the same Economic Times RSS feed at the same moment triggered two identical fetches. Same URL, same response, fetched twice. Ten users, ten fetches. The feed was returning byte-identical XML every time and I was hitting it once per person asking.

The fix is obvious once you see it: the data isn’t per-user, so the fetching shouldn’t be either.

Mistake two: cron

The standard answer is a scheduled job. Run it every 30 minutes, fetch every feed once, write the results to a database. Users read from there instead of from the feeds directly. The store is never more than 30 minutes stale, and you’ve broken the link between user count and fetch count.

This was genuinely better. But a scheduler has one blind spot: it doesn’t know if anyone is actually reading.

WealthWire is a personal project with uneven traffic. Indian markets are open from 9:15 a.m. to 3:30 p.m. on weekdays. Outside those hours, and especially late at night or on weekends, there is almost nobody on the site. The cron job didn’t know that. It fired at 2 a.m., at 3 a.m., through every public holiday, through every weekend. Fetching news that nobody was about to read, refreshing a cache nobody was about to hit.

It worked. It was also paying a quiet, permanent tax for freshness that had no readers.

What I do now: refresh on read

The current approach keeps the best part of cron (one fetch serves everyone) but ties it to actual demand instead of a clock.

The data refreshes when someone asks for it, and only if the last fetch is old enough to need refreshing.

There is no scheduler running in the background. Every incoming request checks one thing first: is the cached copy still fresh? If yes, it gets served immediately with no further work. If no, one request fetches the feeds and writes a new copy. Everyone else in that same moment gets the slightly older copy, and a few seconds later the fresh one is there for them too.

How the layers fit together

A single request passes through three caches before anything actually hits an RSS feed. Each one is cheaper to serve from than the one after it:

Browser cache holds the last response in localStorage with a 30-minute TTL. A user refreshing the page within that window never even leaves their own browser.

CDN / edge cache sits between the browser and the server. With stale-while-revalidate and a matching s-maxage, the CDN can serve a cached response to most visitors while revalidating in the background. No request reaches the server at all.

Server cache (a durable key-value store or blob storage) is the last line before an actual feed fetch. The server checks a timestamp: if the cached data is under 30 minutes old, it returns it. If not, it fetches, writes a fresh copy, and returns that.

A request only reaches a real RSS feed URL if it misses every layer in sequence. On a site with any regular traffic, that is a small fraction of requests.

What happens inside a cache miss

When the server cache is stale, three things happen in order:

Freshness check fails. The timestamp says the data is old. The request needs to fetch.

Lock acquisition. Before fetching, the request tries to claim a short-lived lock (around 60 seconds). Only one request can hold it at a time.

Lock won: fetch and write. The request that wins the lock pulls each feed, parses and normalizes the XML into a common format, and writes the result back to the cache with a current timestamp. If one feed fails (a 404, a timeout, malformed XML), that feed is skipped. The others still write. A broken source costs you that source, not the whole page.

Lock lost: serve what’s there. Every other concurrent request that checked the timestamp at the same moment doesn’t wait. It reads whatever is already in the cache, which is at most 30 minutes old, and returns immediately. By the time the next request comes in, the fresh copy is already there.

Why the lock matters more than anything else

Leave the lock out and the pattern breaks at the worst time.

Imagine 200 people opening the site the moment NSE opens for the day. The cache is 31 minutes old. Without a lock, all 200 requests see stale data, all 200 decide to fetch, and some publisher’s RSS endpoint gets 200 simultaneous requests for the same XML file. You have gone from “one fetch per 30 minutes” to “200 fetches in the same second.” The pattern has made things worse than cron, precisely at the moment your traffic is highest.

The lock collapses that to one fetch. The 199 other requests read the cache that was already there, which is 31 minutes old rather than 30. Nobody notices. The publisher’s server sees one request. A few seconds later, fresh data is in the cache for everyone.

The trade-off worth naming

Freshness is now tied to traffic. If nobody visits WealthWire for six hours, the feeds don’t get pulled for six hours. The next visitor will see data that is a bit older than the stated 30-minute TTL, and will wait a moment while a fresh fetch completes.

For a personal project tracking market hours, that is exactly the right trade. There is no one to serve stale data to at 3 a.m., so there is no cost to not refreshing. The cache goes cold when readers do, and warms back up the moment someone shows up.

If freshness genuinely cannot depend on someone visiting (financial compliance, anything with a regulatory clock), this pattern alone isn’t enough. You would want to keep a lightweight scheduled “warm the cache” ping for off-hours, and use this pattern to handle everything during live traffic.

What to watch once it’s running

A few metrics worth tracking to know the pattern is actually working:

Cache hit ratio per layer. Measure browser, CDN, and server hits separately. If everything is hitting the server cache, the CDN config probably needs work. If everything is hitting the CDN, great.

Outbound feed requests per day. This is the number the whole thing exists to shrink. With a 30-minute TTL the theoretical ceiling is 48 batches per day. If you’re near that number, something is missing at the layers in front.

Lock contention rate. How often does a request find the lock already held? Some contention is fine and expected. A lot of it means traffic is consistently bunching at TTL expiry, and you may want a shorter lock timeout or staggered TTLs per feed.

P95 latency on misses. The miss path (feed fetch plus parse plus write) has real latency. It should be bounded by the slowest feed’s timeout. If it keeps growing, a slow publisher is bleeding into the user experience.

Per-feed failure rate. RSS feeds break quietly. A feed that stops responding just looks like no new articles, not an error. Log fetch outcomes per feed or you will lose visibility into what is actually fresh.

~ tl;tr;

Cron solved the right problem (one fetch, many users) but kept doing work when there was no reason to. The fix isn’t a fancier scheduler. It’s removing the scheduler entirely and letting requests decide when freshness is actually needed.

No background workers. No queue. Just a timestamp, a lock, and a cache in the request path. The site does less work when nobody is reading, which is most of the time.

Built by one person. Runs on quiet infrastructure. Serves Indian finance news to anyone who wants it in one place: wealthwire.in

Understanding useContext and Implementing It in Next.js

Pandiyan MuruganLeave a comment

React provides a powerful feature called useContext that simplifies state sharing across components. In this blog, we’ll explore what useContext is and how to use it effectively in a Next.js application.

What is useContext?

useContext is a React hook that allows you to access the value of a Context directly in functional components. Context provides a way to share data—such as themes, user information, or global settings—across the component tree without manually passing props down at every level.

Benefits of useContext

  • Simplifies state management.
  • Eliminates the need for prop drilling.
  • Works seamlessly with React’s functional components.

How to Use useContext in Next.js

Here’s a step-by-step guide to implementing useContext in a Next.js project.

1. Create a Context

First, create a context with a default value.

import { createContext } from 'react';

// Create a Context
export const MyContext = createContext(null);

2. Create a Context Provider

A Context Provider is a wrapper component that provides the context value to its children.

import React, { useState } from 'react';
import { MyContext } from './MyContext';

export const MyProvider = ({ children }) => {
  const [state, setState] = useState('Hello, Context!');

  return (
    <MyContext.Provider value={{ state, setState }}>
      {children}
    </MyContext.Provider>
  );
};

The MyProvider component wraps children components and provides the context value (state and setState) to all of them.

3. Wrap Your Application with the Provider

Next, wrap your Next.js application with the Context Provider. In Next.js, this is typically done in pages/_app.js.

import { MyProvider } from '../path/to/MyProvider';

function MyApp({ Component, pageProps }) {
  return (
    <MyProvider>
      <Component {...pageProps} />
    </MyProvider>
  );
}

export default MyApp;

This ensures the context is available throughout your application.

4. Access Context Using useContext

Finally, you can use the useContext hook to access the context value in any component.

import React, { useContext } from 'react';
import { MyContext } from '../path/to/MyContext';

const MyComponent = () => {
  const { state, setState } = useContext(MyContext);

  return (
    <div>
      <p>Current State: {state}</p>
      <button onClick={() => setState('Updated Context Value')}>
        Update State
      </button>
    </div>
  );
};

export default MyComponent;

Here, state and setState are accessed directly from the context, eliminating the need for prop drilling.

Example Use Case: Theme Context in Next.js

Let’s implement a real-world example where useContext is used to toggle between light and dark themes in a Next.js application.

1. Create a ThemeContext.js File

import { createContext } from 'react';

export const ThemeContext = createContext({
  theme: 'light',
  toggleTheme: () => {},
});

2. Create a ThemeProvider.js File

import React, { useState } from 'react';
import { ThemeContext } from './ThemeContext';

export const ThemeProvider = ({ children }) => {
  const [theme, setTheme] = useState('light');

  const toggleTheme = () => {
    setTheme((prev) => (prev === 'light' ? 'dark' : 'light'));
  };

  return (
    <ThemeContext.Provider value={{ theme, toggleTheme }}>
      {children}
    </ThemeContext.Provider>
  );
};

3. Wrap the App in pages/_app.js

import { ThemeProvider } from '../path/to/ThemeProvider';

function MyApp({ Component, pageProps }) {
  return (
    <ThemeProvider>
      <Component {...pageProps} />
    </ThemeProvider>
  );
}

export default MyApp;

4. Use the Context in a Component

import React, { useContext } from 'react';
import { ThemeContext } from '../path/to/ThemeContext';

const HomePage = () => {
  const { theme, toggleTheme } = useContext(ThemeContext);

  return (
    <div
      style={{
        background: theme === 'light' ? '#fff' : '#333',
        color: theme === 'light' ? '#000' : '#fff',
      }}
    >
      <h1>Current Theme: {theme}</h1>
      <button onClick={toggleTheme}>Toggle Theme</button>
    </div>
  );
};

export default HomePage;

Here, the theme value determines the background and text color, and the toggleTheme function toggles between light and dark modes.

Conclusion

The useContext hook is a powerful tool for managing state and simplifying component communication in React applications. In Next.js, it integrates seamlessly to provide global state management without the need for external libraries. By following this guide, you can efficiently use useContext to enhance your Next.js projects.