Linux

How to Write Hardened systemd Service Units With Timers, Sandboxing, and Restart Policies

The Back Room Tech Staff · Edited by Aseem Kishore
20 min read

The Back Room Tech is reader-supported. We may earn a commission when you buy through links on our site. Learn more.

A systemd filesystem sandbox can block writes even when the service account owns the directory. Add limits in stages to tell sandbox faults from file permission errors. This Linux lab builds a timestamp task, adds bounded retries and access limits, then schedules daily runs.

Use a nonproduction host that already runs systemd. These commands use Ubuntu/Debian account conventions. Check support in your installed systemd package and local manuals.

Check the Linux Host and Define the Service’s Access

You’ll need admin access through a terminal or SSH, local systemd manuals, and standard account tools. You don’t need extra hardware or a network service. A spare Raspberry Pi or virtual machine is enough.

ScenarioRecommended setting or check
Run once per scheduled activationType=oneshot with Restart=no
Retry a failed, retry-safe taskRestart=on-failure with a delay and start limits
Preserve state under a read-only filesystem sandboxA dedicated directory plus ReadWritePaths=
Prove a write restriction worksSuccessful outside-sandbox write, followed by a denied service write
Reverse the latest hardening changeRemove its named drop-in, reload, and retest

1. Record the distribution, systemd version, and workload requirements

Run:

cat /etc/os-release
systemd --version
uname -r
ps -p 1 -o comm=
systemd-detect-virt

Record the distribution release, full systemd version, kernel, and virtualization result. PID 1 should report:

systemd

systemd-detect-virt can return a nonzero status when it finds no virtualization. That alone isn’t a setup failure.

Containers can block the kernel tools needed for sandboxing. Use a virtual machine if your container host blocks the chosen controls.

Terminal output showing Linux distribution release, systemd version, kernel version, PID 1, and virtualization check

Open the installed manuals. Press q to leave each page:

man systemd.service
man systemd.unit
man systemd.exec
man systemd.timer

Check each directive in these manuals before adding it. The linked upstream manuals describe current upstream behavior. They may include options your distribution doesn’t ship.

Define the lab’s needs:

RequirementLab value
Executable/usr/local/libexec/tbrt-lab-task
Runtime/bin/sh and /usr/bin/date
IdentityDedicated tbrt-lab account and group
Persistent writes/var/lib/tbrt-lab/runs.log
ConfigurationNone
CredentialsNone
Network accessNone
Successful completionExit status 0
Execution limitOne minute

For a real workload, add its config paths, credential sources, network destinations, and retry behavior. A failed task may have done some work before it exits. Automatic retries need to be safe.

Build Hardened systemd Service Units From a Working Baseline

2. Create an isolated account and timestamp workload

Check the required commands and executable paths:

command -v useradd
command -v nologin
command -v install
command -v sudo
command -v tee
command -v getent
command -v false
test -x /bin/sh
test -x /usr/bin/date
test -x /usr/sbin/nologin
test -x /usr/bin/false

The test commands are silent on success. If a path doesn’t exist, resolve it first. Use its verified absolute path throughout.

Check for name and path conflicts:

getent passwd tbrt-lab
getent group tbrt-lab
systemctl show tbrt-lab.service -p LoadState -p FragmentPath
systemctl show tbrt-lab.timer -p LoadState -p FragmentPath
sudo ls -ld /etc/systemd/system/tbrt-lab.service.d
sudo ls -ld /etc/systemd/system/tbrt-lab.timer.d
sudo ls -ld /var/lib/tbrt-lab
sudo ls -l /usr/local/libexec/tbrt-lab-task

For a fresh lab, the account lookups should return no entry. The units should report LoadState=not-found, and the lab-specific paths should be absent.

Stop if anything already exists. Choose a different prefix and replace every lab name consistently. Don’t reuse an existing account or unit.

Create the account and directories:

sudo useradd --system --user-group --no-create-home \
  --home-dir /nonexistent --shell /usr/sbin/nologin tbrt-lab

sudo install -d -o tbrt-lab -g tbrt-lab -m 0750 /var/lib/tbrt-lab
sudo install -d -o root -g root -m 0755 /usr/local/libexec

--system creates a system account. --no-create-home skips home directory creation, and nologin blocks normal interactive login.

Create the workload:

# /usr/local/libexec/tbrt-lab-task
sudo tee /usr/local/libexec/tbrt-lab-task >/dev/null <<'EOF'
#!/bin/sh
set -eu
/usr/bin/date -u '+%Y-%m-%dT%H:%M:%SZ' >> /var/lib/tbrt-lab/runs.log
printf '%s\n' 'tbrt-lab completed'
EOF

sudo chown root:root /usr/local/libexec/tbrt-lab-task
sudo chmod 0755 /usr/local/libexec/tbrt-lab-task

The quoted heredoc delimiter keeps the script text literal. Root owns the script so the service account can’t change its own executable.

Test it under the dedicated account:

sudo -u tbrt-lab /usr/local/libexec/tbrt-lab-task
sudo tail -n 1 /var/lib/tbrt-lab/runs.log

Expected completion message:

tbrt-lab completed

The second command should show one UTC timestamp. Its value depends on when you run the command.

3. Write, validate, and run the minimal service

Create the base unit:

# /etc/systemd/system/tbrt-lab.service
sudo tee /etc/systemd/system/tbrt-lab.service >/dev/null <<'EOF'
[Unit]
Description=Harmless TBRT hardening lab task

[Service]
Type=oneshot
User=tbrt-lab
Group=tbrt-lab
ExecStart=/usr/local/libexec/tbrt-lab-task
TimeoutStartSec=1min
EOF

Type=oneshot suits a finite task. TimeoutStartSec=1min caps its runtime. Oneshot services don’t have a start timeout by default.

Use an explicit executable path. ExecStart= doesn’t process shell redirects or pipelines unless you invoke a shell. See systemd.service.

Validate before loading:

sudo systemd-analyze verify /etc/systemd/system/tbrt-lab.service
echo $?

A successful check should return status 0, usually without output. Read any messages, including warnings that don’t prevent loading.

Then run the service:

sudo systemctl daemon-reload
sudo systemctl start tbrt-lab.service
systemctl show tbrt-lab.service \
  -p Type -p ActiveState -p Result -p ExecMainStatus
sudo tail -n 2 /var/lib/tbrt-lab/runs.log
sudo journalctl -u tbrt-lab.service -b -n 20 --no-pager

Expected property values, with ordering determined by your host:

Type=oneshot
ActiveState=inactive
Result=success
ExecMainStatus=0

The log should contain another timestamp, and the journal should contain tbrt-lab completed.

An inactive state is normal after this task finishes. Leave RemainAfterExit disabled so future timer events can run it again.

4. Add bounded retries without creating a success loop

Create a named restart drop-in:

sudo install -d -o root -g root -m 0755 \
  /etc/systemd/system/tbrt-lab.service.d

# /etc/systemd/system/tbrt-lab.service.d/10-restart.conf
sudo tee /etc/systemd/system/tbrt-lab.service.d/10-restart.conf >/dev/null <<'EOF'
[Unit]
StartLimitIntervalSec=10min
StartLimitBurst=3

[Service]
Restart=on-failure
RestartSec=30s
EOF

Start limits belong in [Unit]. Restart policy and delay belong in [Service].

WorkloadRestart choiceTrade-off
Scheduled task that should wait until its next scheduleRestart=noA failed run gets no immediate automatic retry
Retry-safe scheduled taskRestart=on-failureA partial operation may run again
Long-running foreground processOften Restart=on-failureClean exits don’t trigger a restart
Long-running process that must return after clean exits tooConsider Restart=alwaysIntentional clean exits also restart it

Restart=always and Restart=on-success are invalid for Type=oneshot. An explicit systemctl stop doesn’t cause an automatic restart. See restart policy behavior.

Validate and test:

sudo systemd-analyze verify /etc/systemd/system/tbrt-lab.service
sudo systemctl daemon-reload

# Clear lab counters before this deliberate test.
sudo systemctl reset-failed tbrt-lab.service
sudo systemctl start tbrt-lab.service

systemctl show tbrt-lab.service \
  -p Restart -p RestartUSec -p StartLimitIntervalUSec -p StartLimitBurst
sudo tail -n 1 /var/lib/tbrt-lab/runs.log

Expect Restart=on-failure, a 30-second delay, and a burst limit of three starts per ten minutes.

Start limits also count manual starts. Later lab commands reset the counters before each planned test. This keeps earlier successful runs from skewing the result.

Once the service hits the limit, further starts are blocked until the interval passes. The interval expiring does not queue a pending retry; after it passes, a new manual, timer, or socket activation may start the unit again, and restart logic is available again.

Fix the cause, then reset the failed state and request a start. See systemd.unit start limits.

Add Filesystem and Privilege Restrictions Separately

5. Restrict filesystem access and preserve one state directory

Add the filesystem drop-in:

# /etc/systemd/system/tbrt-lab.service.d/20-filesystem.conf
sudo tee /etc/systemd/system/tbrt-lab.service.d/20-filesystem.conf >/dev/null <<'EOF'
[Service]
ProtectSystem=strict
ProtectHome=yes
PrivateTmp=yes
ReadWritePaths=/var/lib/tbrt-lab
UMask=0027
EOF

Validate and repeat the original workload:

sudo systemd-analyze verify /etc/systemd/system/tbrt-lab.service
sudo systemctl daemon-reload
sudo systemctl reset-failed tbrt-lab.service
sudo systemctl start tbrt-lab.service

systemctl show tbrt-lab.service \
  -p Result -p ExecMainStatus -p ProtectSystem \
  -p ProtectHome -p PrivateTmp -p ReadWritePaths -p UMask
sudo tail -n 1 /var/lib/tbrt-lab/runs.log

The timestamp must still advance. If it doesn’t, stop here and inspect the journal before adding more limits.

SettingEffectWhat can break
ProtectSystem=strictMakes most of the filesystem hierarchy read-onlyState, cache, or log writes outside allowed paths
ReadWritePaths=/var/lib/tbrt-labAllows writes there within the filesystem sandboxMissing directories or ordinary permissions still block writes
ProtectHome=yesMakes /home, /root, and /run/user inaccessibleHome-based configuration, sockets, and credentials
PrivateTmp=yesGives the service private temporary-directory viewsTemporary-file exchange with other services
UMask=0027Restricts permissions on newly created filesConsumers expecting wider access

ReadWritePaths= doesn’t create directories or grant ownership. It also can’t make an underlying read-only filesystem writable.

Only /var/lib/tbrt-lab needs persistent write access here. Private temporary directories remain available through PrivateTmp=yes. Don’t allow all of /var just to solve one failed write.

These settings don’t prevent reading every accessible file or block network access. Those need separate checks. See filesystem namespace settings.

Terminal output showing systemd filesystem sandbox properties alongside a timestamp write and Result=success

6. Prevent privilege gains and restrict kernel interfaces

Add the next set of controls:

# /etc/systemd/system/tbrt-lab.service.d/30-privileges.conf
sudo tee /etc/systemd/system/tbrt-lab.service.d/30-privileges.conf >/dev/null <<'EOF'
[Service]
NoNewPrivileges=yes
ProtectKernelTunables=yes
ProtectKernelModules=yes
ProtectControlGroups=yes
RestrictSUIDSGID=yes
EOF

Retest:

sudo systemd-analyze verify /etc/systemd/system/tbrt-lab.service
sudo systemctl daemon-reload
sudo systemctl reset-failed tbrt-lab.service
sudo systemctl start tbrt-lab.service

systemctl show tbrt-lab.service \
  -p Result -p ExecMainStatus -p NoNewPrivileges \
  -p ProtectKernelTunables -p ProtectKernelModules \
  -p ProtectControlGroups -p RestrictSUIDSGID
sudo tail -n 1 /var/lib/tbrt-lab/runs.log

Expect another successful timestamp write and enabled properties.

NoNewPrivileges=yes prevents the process and its child processes from gaining privileges through execution. Programs that need setuid helpers or file capabilities can fail.

The kernel controls restrict access to kernel settings, module operations, and control-group management. That suits a timestamp task. A host-management agent needs a different review.

For a real service:

  • Move required config out of hidden home directories into a suitable service config path.
  • Grant only the read access required for config.
  • Use a credential mechanism supported by the installed systemd version.
  • Keep secrets out of unit text, command arguments, logs, and screenshots.

I’d skip stronger memory and namespace limits unless the workload supports them. MemoryDenyWriteExecute= can break JIT runtimes. RestrictNamespaces= can break programs that create their own sandboxes.

Network limits need another test stage. PrivateNetwork= or address-family limits can block required connections. The local lab needs no network dependency, but these filesystem controls don’t enforce a network ban.

Schedule the Hardened Task and Check Timer Behavior

7. Validate the calendar before enabling the timer

Check the host clock and how systemd reads the calendar expression:

timedatectl status
systemd-analyze calendar daily

daily uses the host’s local timezone unless the expression specifies one. Confirm that timezone before choosing your schedule.

To inspect an explicit UTC alternative:

systemd-analyze calendar '*-*-* 00:00:00 UTC'

Keep the timer and the checked expression consistent. Calendar syntax is documented in systemd.time.

Create the daily timer:

# /etc/systemd/system/tbrt-lab.timer
sudo tee /etc/systemd/system/tbrt-lab.timer >/dev/null <<'EOF'
[Unit]
Description=Daily TBRT hardening lab task

[Timer]
OnCalendar=daily
Persistent=true
RandomizedDelaySec=15min
Unit=tbrt-lab.service

[Install]
WantedBy=timers.target
EOF

Validate both files and immediately check the result:

sudo systemd-analyze verify \
  /etc/systemd/system/tbrt-lab.service \
  /etc/systemd/system/tbrt-lab.timer
echo $?

Expect status 0 after resolving relevant errors and warnings.

Terminal output showing systemd-analyze verify command on the service and timer units with exit status zero

Enable and start the timer:

sudo systemctl daemon-reload
sudo systemctl enable --now tbrt-lab.timer
systemctl is-enabled tbrt-lab.timer
systemctl is-active tbrt-lab.timer
systemctl list-timers --all tbrt-lab.timer
systemctl status tbrt-lab.timer --no-pager

Expected results:

enabled
active

The timer listing should identify tbrt-lab.service and show its next activation.

Three details affect scheduling:

  • RandomizedDelaySec=15min adds a random delay to spread scheduled work.
  • AccuracySec= lets systemd group timer events. Its documented default is one minute.
  • Persistent=true can cause a catch-up activation when a calendar event was missed while the timer was inactive.

Persistent catch-up doesn’t replay every missed occurrence. It also doesn’t track successful completion. A reconciliation task must find its own outstanding work.

If the service is already active when the timer fires, systemd doesn’t launch another instance of that same service. This includes a oneshot left active with RemainAfterExit=yes. See systemd.timer.

Terminal output showing host timezone, systemd-analyze calendar daily output, and active timer status

Inspect Effective Settings and Prove the Sandbox Works

8. Inspect the loaded files, properties, and security assessment

Read the combined config:

systemctl cat tbrt-lab.service
systemctl cat tbrt-lab.timer

systemctl show tbrt-lab.service \
  -p FragmentPath -p DropInPaths -p User -p Group \
  -p Restart -p RestartUSec \
  -p StartLimitIntervalUSec -p StartLimitBurst \
  -p ProtectSystem -p ProtectHome -p ReadWritePaths \
  -p NoNewPrivileges -p Result -p ExecMainStatus

sudo journalctl -u tbrt-lab.service -b -n 40 --no-pager
sudo systemd-analyze security tbrt-lab.service

systemctl cat identifies the base file and named drop-ins. After daemon-reload, use systemctl show to inspect selected loaded properties. See systemctl.

Expect these service drop-ins:

10-restart.conf
20-filesystem.conf
30-privileges.conf

The security assessment flags exposure based on the settings. It doesn’t prove the program is secure or that each limit works at runtime. Use it to find controls worth reviewing, then test them. See systemd-analyze.

Terminal output showing systemctl cat for the base service and three drop-in files with effective properties
Terminal output showing systemd-analyze security score and sandboxing table for the hardened lab service

9. Compare a permitted host write with a denied sandbox write

Pause scheduling before causing a deliberate failure:

sudo systemctl disable --now tbrt-lab.timer
sudo systemctl stop tbrt-lab.service
sudo ls -ld /var/lib/tbrt-lab-deny-test

The disposable probe directory must be absent. If it exists, stop and choose another unique path throughout this test.

Create it with permissions that allow the service account to write:

sudo install -d -o tbrt-lab -g tbrt-lab -m 0750 \
  /var/lib/tbrt-lab-deny-test

sudo -u tbrt-lab /bin/sh -c \
  'printf "%s\n" outside-ok > /var/lib/tbrt-lab-deny-test/outside.txt'

sudo cat /var/lib/tbrt-lab-deny-test/outside.txt

Expected baseline:

outside-ok

This shows that normal ownership and directory permissions permit the write.

Now add a temporary service probe:

# /etc/systemd/system/tbrt-lab.service.d/90-deny-test.conf
sudo tee /etc/systemd/system/tbrt-lab.service.d/90-deny-test.conf >/dev/null <<'EOF'
[Service]
Restart=no
ExecStartPre=/bin/sh -c 'echo sandbox-probe > /var/lib/tbrt-lab-deny-test/inside.txt'
EOF

sudo systemd-analyze verify /etc/systemd/system/tbrt-lab.service
sudo systemctl daemon-reload
sudo systemctl reset-failed tbrt-lab.service
sudo systemctl start tbrt-lab.service

The start is expected to fail. Restart=no prevents repeated probe attempts.

Inspect the actual result:

systemctl show tbrt-lab.service -p Result -p ExecMainStatus
sudo journalctl -u tbrt-lab.service -b -n 30 --no-pager
sudo ls -la /var/lib/tbrt-lab-deny-test

Expect the shell’s attempted write to fail because the target is read-only inside the service. The inside.txt marker should be absent.

A generic permission error alone doesn’t prove ProtectSystem enforcement. Check the actual journal message. A namespace setup failure means the probe never reached the intended write.

If the outside write succeeds and the inside write fails on a read-only target, unchanged ownership helps isolate the cause. That points to the filesystem sandbox. This test proves the limit for this operation. It doesn’t test every directive.

Terminal output showing read-only filesystem write denial inside the sandboxed service unit

Warning: The following commands delete only the named disposable probe files. Inspect their contents first.

sudo systemctl stop tbrt-lab.service
sudo rm /etc/systemd/system/tbrt-lab.service.d/90-deny-test.conf
sudo rm /var/lib/tbrt-lab-deny-test/outside.txt
sudo rmdir /var/lib/tbrt-lab-deny-test

sudo systemctl daemon-reload
sudo systemd-analyze verify /etc/systemd/system/tbrt-lab.service
sudo systemctl reset-failed tbrt-lab.service
sudo systemctl start tbrt-lab.service
sudo tail -n 1 /var/lib/tbrt-lab/runs.log

The original timestamp task must succeed again. If an unexpected file prevents rmdir, inspect it instead of switching to recursive deletion.

10. Verify failed-run retries and start-rate limiting

Keep the timer disabled. Replace only the executable through another temporary drop-in:

# /etc/systemd/system/tbrt-lab.service.d/90-failure-test.conf
sudo tee /etc/systemd/system/tbrt-lab.service.d/90-failure-test.conf >/dev/null <<'EOF'
[Service]
ExecStart=
ExecStart=/usr/bin/false
EOF

sudo systemd-analyze verify /etc/systemd/system/tbrt-lab.service
sudo systemctl daemon-reload
sudo systemctl reset-failed tbrt-lab.service
sudo systemctl start tbrt-lab.service

The empty ExecStart= clears the original command before setting its replacement. The initial start is expected to fail.

Wait about 100 seconds, then inspect:

sudo journalctl -u tbrt-lab.service -b --since '-3 minutes' --no-pager
systemctl show tbrt-lab.service \
  -p ActiveState -p SubState -p Result -p NRestarts \
  -p Restart -p RestartUSec -p StartLimitBurst

With clean counters, three starts are allowed within ten minutes. Retries are separated by 30 seconds. The next attempted start should hit the limit.

Use the real journal timestamps to confirm the behavior. Don’t infer the number of executed attempts from one restart-counter property alone.

Restore the workload:

Warning: This removes the named failure-test override and stops any pending automatic retry.

sudo systemctl stop tbrt-lab.service
sudo rm /etc/systemd/system/tbrt-lab.service.d/90-failure-test.conf
sudo systemctl daemon-reload
sudo systemd-analyze verify /etc/systemd/system/tbrt-lab.service
sudo systemctl reset-failed tbrt-lab.service
sudo systemctl start tbrt-lab.service

systemctl show tbrt-lab.service -p ActiveState -p Result -p ExecMainStatus
sudo tail -n 1 /var/lib/tbrt-lab/runs.log

Expect one successful completion followed by ActiveState=inactive. Successful oneshots shouldn’t enter a restart loop.

11. Observe persistent catch-up on a short lab schedule

Use a temporary timer override:

sudo install -d -o root -g root -m 0755 \
  /etc/systemd/system/tbrt-lab.timer.d

# /etc/systemd/system/tbrt-lab.timer.d/90-catchup-test.conf
sudo tee /etc/systemd/system/tbrt-lab.timer.d/90-catchup-test.conf >/dev/null <<'EOF'
[Timer]
OnCalendar=
OnCalendar=*-*-* *:*:00
RandomizedDelaySec=0
AccuracySec=1s
EOF

systemd-analyze calendar '*-*-* *:*:00'
sudo systemd-analyze verify /etc/systemd/system/tbrt-lab.timer
sudo systemctl daemon-reload
sudo systemctl reset-failed tbrt-lab.service
sudo systemctl start tbrt-lab.timer
systemctl list-timers --all tbrt-lab.timer

The empty OnCalendar= clears the daily expression. The replacement schedules activation at each minute boundary.

Wait until one scheduled run completes, then inspect and stop the timer:

sudo journalctl -u tbrt-lab.service --since '-2 minutes' --no-pager
sudo tail -n 3 /var/lib/tbrt-lab/runs.log
sudo systemctl stop tbrt-lab.timer
date -u

Leave it stopped across at least two minute boundaries. Then restart it:

sudo systemctl reset-failed tbrt-lab.service
sudo systemctl start tbrt-lab.timer
systemctl list-timers --all tbrt-lab.timer
sudo journalctl -u tbrt-lab.service --since '-3 minutes' --no-pager
sudo tail -n 5 /var/lib/tbrt-lab/runs.log

Look for one catch-up activation after restarting. Check promptly because normal minute-based runs will resume too.

Restore the daily schedule:

Warning: This deletes the temporary timer override and restarts the timer with its daily schedule.

sudo systemctl stop tbrt-lab.timer
sudo systemctl stop tbrt-lab.service
sudo rm /etc/systemd/system/tbrt-lab.timer.d/90-catchup-test.conf
sudo rmdir /etc/systemd/system/tbrt-lab.timer.d

sudo systemctl daemon-reload
sudo systemd-analyze verify \
  /etc/systemd/system/tbrt-lab.service \
  /etc/systemd/system/tbrt-lab.timer
sudo systemctl reset-failed tbrt-lab.service
sudo systemctl enable --now tbrt-lab.timer
systemctl cat tbrt-lab.timer
systemctl list-timers --all tbrt-lab.timer

Confirm that OnCalendar=daily and the 15-minute randomized delay are restored.

Fix the Failing Layer and Roll Back One Change

12. Match common failures to a specific check

SymptomLikely layerTargeted action
Unknown directive or ignored settingSyntax or installed-version supportRead the local manual, correct its section, verify, and reload
State write failsOwnership or filesystem sandboxInspect the directory and effective writable-path allowance
Configuration disappearsHome-directory isolation or access permissionsIdentify its exact path and provide a suitable service configuration path
Retries stopStart-rate limitFix the cause, reset counters, and start deliberately
Timer is enabled but inactiveActivation stateStart it with enable --now
Later timer activations do nothingService remains activeInspect state and remove unintended RemainAfterExit=yes

For state-write failures, inspect both layers:

sudo ls -ld /var/lib/tbrt-lab
sudo ls -l /var/lib/tbrt-lab/runs.log
systemctl show tbrt-lab.service -p ProtectSystem -p ReadWritePaths
sudo journalctl -u tbrt-lab.service -b -n 30 --no-pager

The directory should belong to tbrt-lab:tbrt-lab. Correct only this lab directory if its ownership is wrong:

sudo chown tbrt-lab:tbrt-lab /var/lib/tbrt-lab
sudo chmod 0750 /var/lib/tbrt-lab

If the existing log has incorrect ownership, inspect it and correct that file separately. Avoid recursive permission changes.

For an invalid oneshot restart policy, edit the known restart drop-in:

sudoedit /etc/systemd/system/tbrt-lab.service.d/10-restart.conf

Use Restart=no or Restart=on-failure, then verify and reload. Don’t move start-limit settings into [Service].

For an inactive timer:

sudo systemctl enable --now tbrt-lab.timer
systemctl status tbrt-lab.timer --no-pager

For unexpected service activity:

systemctl show tbrt-lab.service \
  -p ActiveState -p SubState -p RemainAfterExit
systemctl cat tbrt-lab.service

Inspect a running task before requesting another activation. A completed recurring oneshot shouldn’t be held active by RemainAfterExit=yes.

13. Remove the last hardening increment safely

If 30-privileges.conf caused the failure, roll back that file alone.

Warning: This stops the lab schedule and service, then removes the named privilege-hardening drop-in.

sudo systemctl disable --now tbrt-lab.timer
sudo systemctl stop tbrt-lab.service

sudo cat /etc/systemd/system/tbrt-lab.service.d/30-privileges.conf
sudo rm /etc/systemd/system/tbrt-lab.service.d/30-privileges.conf

sudo systemctl daemon-reload
sudo systemd-analyze verify /etc/systemd/system/tbrt-lab.service
sudo systemctl reset-failed tbrt-lab.service
sudo systemctl start tbrt-lab.service

systemctl show tbrt-lab.service -p Result -p ExecMainStatus
sudo tail -n 1 /var/lib/tbrt-lab/runs.log

Expect success under the previous filesystem-hardening stage. Only then restore scheduling:

sudo systemctl enable --now tbrt-lab.timer

This rollback leaves the privilege controls removed. To restore them, recreate the exact file from step 6 and repeat its checks.

For complete lab removal, first stop both units. Remove only the named unit files, remaining known drop-ins, and workload script. Reload systemd afterward. Save any state you want before deleting the dedicated directory, account, and group.

Avoid broad wildcard deletion. A named drop-in keeps the rollback small enough to review.

14. Record acceptance checks and verify reboot persistence

Before adapting this design for production, record:

  • Distribution, systemd package version, kernel, and supported directives.
  • Executable ownership, service identity, config paths, and credential access.
  • Required writable directories and successful writes from within the service.
  • Actual denied-operation results and failure reasons.
  • Retry safety, delay, and observed start-limit behavior.
  • Timezone, scheduling tolerance, catch-up expectations, and overlap behavior.
  • Effective files, relevant logs, and exact rollback paths.

Keep this lab separate from existing production units. Apply reviewed changes during a suitable maintenance window.

Before rebooting, confirm the timer is enabled:

systemctl is-enabled tbrt-lab.timer
systemctl list-timers --all tbrt-lab.timer

Warning: The next command reboots the lab host and disconnects SSH. Save your work first.

sudo systemctl reboot

Reconnect, then check:

systemctl is-enabled tbrt-lab.timer
systemctl is-active tbrt-lab.timer
systemctl list-timers --all tbrt-lab.timer

sudo systemctl reset-failed tbrt-lab.service
sudo systemctl start tbrt-lab.service
systemctl show tbrt-lab.service -p Result -p ExecMainStatus
sudo tail -n 1 /var/lib/tbrt-lab/runs.log

The timer should be enabled and active after boot. The manual start separately proves that the hardened workload can still write its state.

Wrapping Up

A dedicated account, explicit write allowance, and tested restart policy make faults easier to trace. I’d keep each hardening group in its own drop-in. A smaller rollback is worth the extra files.

StepActionApplies To
Establish a baselineVerify identity and successful executionEvery custom service
Add restrictionsRetest required writes after each incrementHardened services
Prove behaviorTest denial, retries, and catch-upThis lab and reviewed adaptations
Check persistenceConfirm timer activity and a successful write after rebootScheduled deployment

After reboot, confirm the timer is active and the hardened service appends a fresh timestamp. That’s the final acceptance check.