A task that looks simple at the application level:

Website → Detect technologies → Return JSON

may actually execute something closer to:

Application
    ↓
Docker CLI
    ↓
Docker Engine
    ↓
Wappalyzer container
    ↓
Playwright
    ↓
Chromium
    ├── Browser process
    ├── Renderer processes
    ├── GPU process
    └── Utility processes

I recently investigated a situation where Wappalyzer-based technology detection gradually caused high CPU and memory usage on a server.

The individual scans were supposed to have strict timeouts.

The Docker containers were also started with --rm.

Yet some containers survived far longer than expected, leaving Chromium processes consuming significant resources.

The interesting part was that the problem wasn't simply:

Chromium uses a lot of memory.

The real issue was lifecycle management.

This article explains the problem, why the obvious timeout implementation wasn't enough, and how to make this kind of workload much safer.

The Architecture

Each technology-detection request launched Wappalyzer inside its own Docker container.

A simplified command looked like this:

docker run --rm \
  wappalyzer-image \
  -i https://example.com \
  --scan-type full \
  -t 30 \
  -oJ -

The application launched that command asynchronously:

process = await asyncio.create_subprocess_exec(
    *command,
    stdout=asyncio.subprocess.PIPE,
    stderr=asyncio.subprocess.PIPE,
)

It then waited for the scan to finish:

stdout, stderr = await asyncio.wait_for(
    process.communicate(),
    timeout=30,
)

If the timeout was exceeded:

process.kill()
await process.wait()

At first glance, this looks reasonable.

There is a timeout.

There is process cleanup.

And Docker is using:

--rm

So why could containers remain alive?

The First Sign of Trouble

The first indication was unusually high server resource consumption.

For this kind of issue, docker stats is one of the quickest places to start:

docker stats

It lets you see:

• CPU consumption

• memory consumption

• memory limits

• network activity

• block I/O

• PID counts

The important discovery was that some temporary Wappalyzer containers were consuming significant resources and had been running much longer than expected.

The next step was checking Chromium processes.

For Playwright installations, something like this can be useful:

pgrep -af '/ms-playwright.*chrome'

And if Wappalyzer uses identifiable Chromium profile directories:

pgrep -af 'wappalyzer-chromium'

There were many Chromium processes associated with technology-detection jobs.

Chromium spawning multiple processes is normal.

A single browser instance may create separate processes for renderers, networking, GPU operations, storage, and other browser services.

The abnormal part was the lifetime of those processes.

A scan intended to finish in approximately 30 seconds should not leave a container alive for minutes—or potentially much longer.

Looking at the Host, Not Just Docker

Container metrics only tell part of the story.

I also checked the host using:

vmstat 1

This is particularly useful when diagnosing whether the machine is experiencing:

• CPU saturation

• high runnable process queues

• swapping

• I/O waiting

• virtualization steal time

For example, the important columns include:

r   runnable processes
si  swap in
so  swap out
us  user CPU
sy  system CPU
id  idle CPU
wa  I/O wait
st  stolen CPU

When browser workloads are overwhelming a machine, you may see the runnable queue increase while CPU idle time approaches zero.

That gives a much better picture than looking at application logs alone.

The Important Process Boundary

The root cause becomes clearer when you look at what the application is actually controlling.

The process tree conceptually looks like this:

Python application
        │
        ▼
    docker run
        │
        ▼
   Docker Engine
        │
        ▼
     Container
        │
        ▼
    Wappalyzer
        │
        ▼
    Playwright
        │
        ▼
     Chromium

This distinction matters.

When Python executes:

process = await asyncio.create_subprocess_exec(...)

the subprocess being managed is:

docker run

It is not Chromium.

It is also not the Docker container itself.

The Docker CLI communicates with Docker Engine, which creates and manages the container independently.

Therefore:

process.kill()

means roughly:

Kill the local Docker CLI process that Python started.

It does not provide the stronger guarantee:

Destroy the container associated with this job and terminate everything running inside it.

That difference is easy to miss.

Why docker run --rm Isn't a Watchdog

Another assumption that can cause problems is treating:

docker run --rm

as lifecycle protection.

--rm is useful, but its job is more specific.

Conceptually:

Container exits
      ↓
Docker automatically removes it

It prevents exited temporary containers from accumulating.

But consider this situation:

Python timeout occurs
        ↓
docker CLI process is killed
        ↓
Docker container remains alive
        ↓
Chromium remains alive

The container hasn't exited.

Therefore there is nothing for --rm to remove yet.

So:

--rm is automatic post-exit cleanup, not a container watchdog.

That's an important distinction for dynamically created containers.

Give Every Temporary Container an Identity

The first improvement is simple:

Name every job container.

Generate a unique identifier:

import uuid

container_name = f"wappalyzer-scan-{uuid.uuid4().hex}"

Then launch the container with:

docker run --rm \
  --name wappalyzer-scan-<uuid> \
  ...

Now there is an explicit relationship between the application job and the Docker resource:

Application job
      ↕
Container name

This becomes extremely useful when something goes wrong.

Instead of trying to discover which anonymous Docker container belongs to the failed scan, the application already knows exactly what to remove.

Use Layered Timeouts

Another problem was having the internal Wappalyzer timeout and the application watchdog effectively use the same value.

For example:

Wappalyzer timeout: 30 seconds
Application timeout: 30 seconds

That creates a race.

At approximately 30 seconds, Wappalyzer may be trying to:

finish the scan
    ↓
collect results
    ↓
serialize JSON
    ↓
close Playwright
    ↓
terminate Chromium
    ↓
exit

At exactly the same time, the parent process may decide:

TIMEOUT

and kill the Docker CLI.

A cleaner model is to separate the two responsibilities.

For example:

Internal scan timeout: 30 seconds
Hard watchdog:         45 seconds

The internal timeout controls normal Wappalyzer behavior.

The outer timeout protects the application from a container that fails to terminate correctly.

In Python:

scan_timeout = 30
hard_timeout_grace = 15

hard_timeout = scan_timeout + hard_timeout_grace

Then:

stdout, stderr = await asyncio.wait_for(
    process.communicate(),
    timeout=hard_timeout,
)

Now Wappalyzer has time to perform normal cleanup before the application applies the hard limit.

Explicitly Destroy Timed-Out Containers

This is the most important change.

When the hard timeout occurs, don't only kill the Docker CLI process.

Also tell Docker Engine to remove the corresponding container.

Conceptually:

try:
    stdout, stderr = await asyncio.wait_for(
        process.communicate(),
        timeout=hard_timeout,
    )

except TimeoutError:
    await terminate_process(process)
    await remove_container(container_name)
    raise

The cleanup function can execute:

docker rm -f <container-name>

For example:

async def remove_container(container_name: str) -> None:
    cleanup = await asyncio.create_subprocess_exec(
        "docker",
        "rm",
        "-f",
        container_name,
        stdout=asyncio.subprocess.DEVNULL,
        stderr=asyncio.subprocess.DEVNULL,
    )

    await asyncio.wait_for(
        cleanup.wait(),
        timeout=10,
    )

This creates a much stronger guarantee.

Instead of saying:

Stop waiting for docker run.

the application now says:

This job exceeded its maximum lifetime.

Destroy its container.

When Docker force-removes the container, the Chromium process tree inside it is terminated as well.

Handle Async Cancellation Too

Timeout isn't the only abnormal exit path.

An asynchronous task can also be cancelled because of:

• application shutdown

• worker shutdown

• orchestration changes

• task cancellation

• deployment

• upstream failure

So cleanup should also handle cancellation.

For example:

except (TimeoutError, asyncio.CancelledError):
    await terminate_process(process)
    await remove_container(container_name)
    raise

The goal is to maintain this invariant:

Application job no longer exists
             =
Temporary job container no longer exists

Without cancellation cleanup, you can fix timeout leaks while still leaving another path for orphaned containers.

Add CPU Limits

Even with perfect cleanup, browser automation can be CPU intensive.

A problematic website can cause Chromium to perform expensive rendering, JavaScript execution, network processing, or browser work.

Without resource limits:

docker run ...

the container can potentially consume substantial host CPU.

Docker allows you to put a ceiling on this:

--cpus 1

For example:

docker run \
  --cpus 1 \
  ...

This doesn't mean one CPU is the correct value for every environment.

The right value depends on:

• server hardware

• expected scan duration

• concurrency

• target websites

• acceptable throughput

But the principle is important:

One technology-detection job shouldn't automatically have unrestricted access to host CPU resources.

Start with a conservative limit and benchmark it.

Add Memory Limits

The same principle applies to memory.

Chromium can consume significant RAM, especially on complex websites.

Docker can enforce a memory ceiling:

--memory 1g

You can also restrict the container from consuming additional swap beyond that allocation:

--memory-swap 1g

Combined:

docker run --rm \
  --name wappalyzer-scan-<uuid> \
  --memory 1g \
  --memory-swap 1g \
  --cpus 1 \
  wappalyzer-image ...

The 1g value here should be treated as an example starting point.

It should be tested against real workloads.

If legitimate scans repeatedly hit the memory ceiling, increase it based on measurements rather than removing the limit entirely.

The objective is containment.

Without limits:

Bad scan
    ↓
Chromium consumes more resources
    ↓
Host becomes unstable

With limits:

Bad scan
    ↓
Container reaches resource boundary
    ↓
Impact remains contained

Resource Limits Don't Replace Concurrency Control

Per-container limits solve one part of the problem.

Concurrency solves another.

Suppose each container is allowed:

1 CPU
1 GB RAM

One container has a predictable maximum impact.

But if the application launches:

20 containers

the aggregate workload can still overwhelm the host.

A useful way to think about capacity is:

Maximum concurrent jobs
        ×
Per-job resource ceiling
        ≈
Maximum browser workload

For example:

2 concurrent jobs × 1 GB
≈ 2 GB potential browser memory

versus:

10 concurrent jobs × 1 GB
≈ 10 GB potential browser memory

The limits don't mean every container will consume its maximum allocation, but they give you a useful upper bound.

So production browser workloads generally need both:

Per-job resource limits
          +
Concurrency control

Monitor Temporary Containers

Short-lived containers can be difficult to observe because they may appear and disappear between normal docker ps commands.

A simple watcher helps:

watch -n 0.5 \
  'docker ps --filter ancestor=wappalyzer-image \
  --format "table {{.Names}}\t{{.Status}}\t{{.RunningFor}}"'

During normal operation, you should see:

wappalyzer-scan-a1b2c3...

Then a few seconds later:

<container disappears>

That is healthy behavior.

The lifecycle should look like:

Container created
      ↓
Wappalyzer starts
      ↓
Chromium starts
      ↓
Detection completes
      ↓
Chromium exits
      ↓
Container exits
      ↓
--rm removes container

What you don't want is:

Container created
      ↓
Expected timeout passes
      ↓
Hard watchdog passes
      ↓
Container still running

If that happens, the cleanup mechanism needs investigation.

Monitor Application-Level Events Too

Docker monitoring tells you what's happening at the infrastructure layer.

Application logs tell you whether scans are actually progressing.

For example, logging events such as:

technology_detection_started
technology_detection_completed

makes it much easier to correlate:

Application job
      ↕
Docker container
      ↕
Website scan

A scan might produce:

technology_detection_started
...
technology_detection_completed

within 10–30 seconds.

That is very different from seeing a Docker container alive for several minutes without a corresponding active application task.

Combining application logs with Docker-level monitoring is much more useful than relying on either one alone.

Verify the Docker Limits

Don't assume that because your application constructed the command correctly, Docker actually received the expected configuration.

Inspect a running scan:

docker inspect wappalyzer-scan-<uuid> \
  --format 'Memory={{.HostConfig.Memory}} MemorySwap={{.HostConfig.MemorySwap}} NanoCpus={{.HostConfig.NanoCpus}}'

You can also inspect real-time resource consumption:

docker stats --no-stream wappalyzer-scan-<uuid>

This verifies the configuration at the actual container boundary.

For infrastructure problems, verification at the infrastructure layer is important.

Test the Failure Path

Lifecycle code is most important when something fails.

So don't only test successful Wappalyzer responses.

Useful test cases include:

Successful scan
Default timeout
Custom timeout
Provider failure
Command construction
Hard timeout
Task cancellation
Container cleanup

For example, the timeout test can deliberately make:

process.communicate()

take longer than the hard watchdog.

Then verify that both operations occur:

Docker CLI subprocess terminated
              +
Docker container explicitly removed

Cancellation should receive similar coverage.

This is especially important because orphan-container bugs often don't appear during ordinary happy-path testing.

A Safer Implementation

Putting the pieces together, the provider can follow a structure similar to this:

import asyncio
import uuid


class TechnologyDetector:
    def __init__(
        self,
        image: str,
        timeout_seconds: int = 30,
        hard_timeout_grace_seconds: int = 15,
        memory_limit: str = "1g",
        cpu_limit: str = "1",
    ):
        self.image = image
        self.timeout_seconds = timeout_seconds
        self.hard_timeout_grace_seconds = hard_timeout_grace_seconds
        self.memory_limit = memory_limit
        self.cpu_limit = cpu_limit


    async def detect(self, website: str) -> str:
        scan_timeout = self.timeout_seconds

        hard_timeout = (
            scan_timeout
            + self.hard_timeout_grace_seconds
        )

        container_name = (
            f"wappalyzer-scan-{uuid.uuid4().hex}"
        )

        command = [
            "docker",
            "run",
            "--rm",
            "--name",
            container_name,
            "--memory",
            self.memory_limit,
            "--memory-swap",
            self.memory_limit,
            "--cpus",
            self.cpu_limit,
            self.image,
            "-i",
            website,
            "--scan-type",
            "full",
            "-t",
            str(scan_timeout),
            "-oJ",
            "-",
        ]

        process = await asyncio.create_subprocess_exec(
            *command,
            stdout=asyncio.subprocess.PIPE,
            stderr=asyncio.subprocess.PIPE,
        )

        try:
            stdout, stderr = await asyncio.wait_for(
                process.communicate(),
                timeout=hard_timeout,
            )

        except (TimeoutError, asyncio.CancelledError):
            await self._terminate_process(process)
            await self._remove_container(container_name)
            raise

        if process.returncode != 0:
            raise RuntimeError(
                stderr.decode(errors="replace")
            )

        return stdout.decode()


    @staticmethod
    async def _terminate_process(process) -> None:
        if process.returncode is not None:
            return

        try:
            process.kill()
        except ProcessLookupError:
            return

        await process.wait()


    async def _remove_container(
        self,
        container_name: str,
    ) -> None:
        try:
            cleanup = await asyncio.create_subprocess_exec(
                "docker",
                "rm",
                "-f",
                container_name,
                stdout=asyncio.subprocess.DEVNULL,
                stderr=asyncio.subprocess.DEVNULL,
            )

            await asyncio.wait_for(
                cleanup.wait(),
                timeout=10,
            )

        except (
            TimeoutError,
            FileNotFoundError,
            OSError,
        ):
            return

This isn't meant to be a drop-in library implementation.

The important concepts are the lifecycle guarantees:

Unique container identity
        +
Internal timeout
        +
Hard watchdog
        +
Explicit container cleanup
        +
CPU limit
        +
Memory limit
        +
Cancellation cleanup

Before and After

The difference can be summarized simply.

Before

Application
    ↓
docker run --rm
    ↓
Wappalyzer
    ↓
Chromium

30-second timeout
    ↓
Kill docker CLI
    ↓
Assume everything disappeared

The dangerous word there is:

Assume

After

Application
    ↓
Generate unique container name
    ↓
Start resource-limited container
    ↓
Wappalyzer
    ↓
Playwright / Chromium

Internal timeout
    ↓
Grace period
    ↓
Hard watchdog
    ↓
Terminate CLI if necessary
    ↓
docker rm -f <known-container>

Now cleanup is explicit.

The Broader Lesson

This problem isn't specific to Wappalyzer.

The same principle applies to workloads involving:

• Playwright

• Puppeteer

• Selenium

• headless Chromium

• browser-based crawlers

• screenshot services

• PDF rendering

• automated testing

• scraping infrastructure

The important realization is that there are multiple lifecycle boundaries:

Application task
      ↓
Subprocess
      ↓
Docker container
      ↓
Browser process tree

Controlling one does not automatically mean you control the others.

A Python subprocess timeout controls the subprocess.

A Docker container limit controls the container.

A browser timeout controls browser operations.

Each boundary needs to be considered explicitly.

Practical Checklist

If you're running Wappalyzer or another browser-heavy workload inside temporary Docker containers, check the following:

• Give every temporary container a unique name.

• Keep --rm, but don't rely on it as a watchdog.

• Set an internal scan timeout.

• Give the outer watchdog additional cleanup time.

• Explicitly docker rm -f containers that exceed the hard timeout.

• Handle async cancellation as well as timeout.

• Add CPU limits.

• Add memory limits.

• Limit overall concurrency.

• Monitor Docker resource consumption.

• Monitor browser process counts.

• Correlate application events with container lifetime.

• Test timeout and cancellation paths.

• Verify Docker limits with docker inspect.

• Benchmark limits against real websites before deploying them broadly.

Final Takeaway

The biggest lesson from this issue was simple:

A subprocess timeout, a Docker container lifetime, and a browser lifetime are not the same thing.

When an application dynamically creates browser containers, it should know exactly which container belongs to each job and be capable of explicitly destroying that container when the job ends abnormally.

Resource limits provide the second layer of protection.

Even if cleanup fails, a single browser job shouldn't be able to consume unrestricted CPU or memory.

The combination of:

Unique identity
+ resource limits
+ layered timeouts
+ explicit cleanup
+ concurrency control
+ monitoring

turns a potentially unpredictable browser workload into something much easier to operate safely.

And one final lesson worth remembering:

docker run --rm is cleanup convenience. It is not lifecycle management.