One of the most common "bizarre" issues encountered after restarting a Linux server—especially in production environments running complex application stacks—is the incorrect startup sequence of services.

You might find that an application server (such as Tomcat or Node.js) has finished starting up, but the underlying MySQL database is still initializing, causing the application to crash due to a "connection refused" error. Alternatively, Nginx might take control of port 80 before the backend API it is proxying is ready, resulting in "502 Bad Gateway" errors.

The root cause of these problems lies in a conflict between Systemd’s default parallel startup mechanism and implicit dependencies. Drawing on troubleshooting logic from professional operations (Ops) blogs, this article will guide you through everything from the underlying principles to practical configuration, showing you how to strictly enforce the startup order of your Linux services.

Core Principle: Systemd Parallelism and Dependencies

Older systems using SysVinit relied on a strictly sequential startup process, governed by numbered scripts (e.g., S01, S02). In contrast, modern Linux distributions (such as Ubuntu, CentOS, and Debian) utilize Systemd. Its primary advantage is parallel startup, which maximizes multi-core CPU utilization and reduces boot times.

However, for parallel startup to function correctly, Systemd must have a precise understanding of the dependencies between services. If you define a custom service in a `.service` file but specify only the `ExecStart` command without declaring any dependencies, Systemd assumes the service can be launched at any time. Consequently, the system merely issues the startup command without ensuring that the underlying services are actually ready.

Diagnosis: Identifying the Culprit Behind Startup Issues

Before modifying any configurations, use the following commands to gather evidence and pinpoint which services are "jumping the gun."

1. Analyze the Critical Chain

Run the following command to display a tree-like view of the dependency chain that took the longest to complete during the boot process:

systemd-analyze critical-chain

Look for the timestamps marked with an `@` symbol in the output. If you find that the timestamp for `my-app.service` precedes that of `mysql.service`, and there is no `+` sign connecting them (which would indicate a dependency), it means the database was not included in the wait queue when the application started.

2. Check service startup duration (Blame)

Identify services that take an exceptionally long time to start and are dragging down the overall boot process:

systemd-analyze blame

If you notice a service taking an unusually long time (for example, `NetworkManager-wait-online.service` taking 30 seconds), it is likely the culprit behind timeouts in subsequent services that depend on the network.

3. View logs for a specific service

If an application crashes because it cannot connect to the database, you can examine the application's logs:

journalctl -u my-app.service -b

The logs will typically show "Connection refused," confirming that the application attempted to start before the database was ready.

Practical Fix: Enforcing Startup Order

Suppose you have an application service, `my-app.service`, which must start only after `mysql.service` and `network-online.target`. You need to edit the service's unit file:

sudo systemctl edit --full my-app.service

In the `[Unit]` section, add the following parameters:

ini

[Unit]

Description=My Application Service

# 1. Ensure the network is truly available (not just that the network interface has been initialized)

After=network-online.target mysql.service

# 2. Define a strong dependency: if MySQL fails to start, this service will not start

Requires=mysql.service

# 3. Define a weak dependency: if there is a network delay, this service will still attempt to start

Wants=network-online.target

Key parameter explanations:

`After`: Defines the startup order, ensuring that `my-app` starts only after `mysql`.

`Requires`: Defines a strong dependency. If `mysql` fails or stops, `my-app` will also be stopped; if `mysql` fails to start, `my-app` will not start.

`Wants`: Defines a weak dependency. It is recommended to set dependencies such as network and peripheral access as `Wants` rather than `Requires`. This prevents local services from becoming completely deadlocked due to transient network fluctuations.

`network-online.target`: This is a common pitfall. While `network.target` merely indicates that network services (such as NetworkManager) have started, `network-online.target` signifies that network interfaces have successfully acquired an IP address and configured routing. Applications that require network access to fetch data must depend on the latter.

After saving your changes, run the following commands to apply the configuration and restart the service:

sudo systemctl daemon-reload

sudo systemctl restart my-app.service

Handling Special Cases: Mount Points and Latency

Sometimes, a service may depend on a specific data disk being mounted (e.g., `/data`). If the disk mount is slow, the service might start prematurely—reading an empty directory and causing initialization to fail.

Managing Mount Dependencies:

Add `RequiresMountsFor=/data` to the `[Unit]` section. This automatically infers a dependency on the `.mount` unit for the mount point containing `/data`, ensuring the disk is fully mounted before your service starts.

Handling Startup Failures with Retries:

Even with dependencies configured, a prerequisite service might start, yet still require tens of seconds for internal "warm-up" (such as InnoDB recovery in MySQL). Rather than indefinitely extending Systemd’s wait time, you can configure automatic retries in the `[Service]` section:

ini

[Service]

Restart=on-failure

RestartSec=10

This ensures that if the application crashes or exits because a prerequisite service is not yet ready, the system will automatically attempt to restart it after 10 seconds. Typically, after two or three attempts, the underlying database will be fully ready.

Why Is Startup Speed ​​Tied to the Underlying Hardware?

Troubleshooting and fixing startup sequence issues is, in many ways, a race against a "time window."

If a server uses inexpensive shared storage, a "cold start" of a database—including InnoDB data recovery—can take over a minute, significantly widening the "gap" during which dependencies remain unfulfilled. If cross-border network connections suffer from congestion or packet loss, services that need to fetch remote configurations or images during startup may time out—potentially causing the entire boot sequence to collapse.

This is where Jtti provides a foundational guarantee for business stability:

Ultra-fast cold starts: Every Jtti instance comes equipped with enterprise-grade NVMe SSDs. Their exceptionally high random read/write IOPS allow I/O-intensive services—such as MySQL—to load data rapidly upon restart, significantly reducing the time required for core services to become operational.

Deterministic network paths: Jtti’s Hong Kong and U.S. nodes utilize premium CN2 GIA network routes, maintaining a packet loss rate of less than 0.5% even during evening peak hours. This ensures that if your services require API handshakes or data synchronization with remote endpoints during startup, connections won’t face indefinite timeouts due to network instability, leading to a much smoother `network-online` phase.

No resource oversubscription: Leveraging KVM-based hardware-level isolation, Jtti ensures that your allocated CPU and memory resources are genuinely available. During a system restart, concurrent processes won’t be throttled by "noisy neighbors" competing for compute power.

Long-term cost predictability: Jtti maintains a "same-price renewal" policy, ensuring that your long-term infrastructure costs remain stable and protecting you from the risk of forced migrations caused by hidden price hikes.

Optimizing boot sequences reflects a high level of operational precision; meanwhile, choosing high-performance, low-latency infrastructure like Jtti sets a solid physical baseline for your startup process. By mastering Systemd dependency logic and pairing it with robust underlying hardware, you ensure that your services can return online as quickly and reliably as possible following any restart. Visit the Jtti official website to explore high-performance cloud server solutions designed for building highly available business stacks.

FAQ

Q1: Why does my application still fail to connect and restart, even though I’ve added `After=mysql.service`?

A: The `After` directive only dictates the *sequence* of actions—it simply signals to Systemd that the application should start after the MySQL service. While the MySQL process itself may have launched, the internal port (e.g., 3306) might not yet be actively listening for connections. The solution is to configure `Restart=on-failure` (as mentioned above) in addition to the `After` directive; this allows the application to automatically wait and attempt to reconnect if it is initially rejected.

Q2: Should I use `Wants` or `Requires`?

A: Use `Requires` if two services are "inextricably linked"—meaning they must coexist (e.g., a frontend application and a local database). Use `Wants` if the services are merely "cooperative," where the failure of one should not necessarily cause the other to fail (e.g., an application that depends on an external network, where you wouldn't want the application to shut down just because the network experiences a temporary hiccup). It is generally recommended to use `Wants` for network dependencies.

Q3: Why do changes to a `.service` file not take effect after restarting the service?

A: Whenever you modify a Systemd unit file, you must run `sudo systemctl daemon-reload` to reload the Systemd configuration. Only then should you execute `sudo systemctl restart <service-name>` to ensure the updated configuration is correctly parsed and applied.