A systemd filesystem sandbox can block writes even when the service account owns the directory. Add limits in stages to tell sandbox faults from file permission errors. This Linux lab builds a timestamp task, adds bounded retries and access limits, then schedules daily runs.
Use a nonproduction host that already runs systemd. These commands use Ubuntu/Debian account conventions. Check support in your installed systemd package and local manuals.
Check the Linux Host and Define the Service’s Access
You’ll need admin access through a terminal or SSH, local systemd manuals, and standard account tools. You don’t need extra hardware or a network service. A spare Raspberry Pi or virtual machine is enough.
| Scenario | Recommended setting or check |
|---|---|
| Run once per scheduled activation | Type=oneshot with Restart=no |
| Retry a failed, retry-safe task | Restart=on-failure with a delay and start limits |
| Preserve state under a read-only filesystem sandbox | A dedicated directory plus ReadWritePaths= |
| Prove a write restriction works | Successful outside-sandbox write, followed by a denied service write |
| Reverse the latest hardening change | Remove its named drop-in, reload, and retest |
1. Record the distribution, systemd version, and workload requirements
Run:
cat /etc/os-release
systemd --version
uname -r
ps -p 1 -o comm=
systemd-detect-virt
Record the distribution release, full systemd version, kernel, and virtualization result. PID 1 should report:
systemd
systemd-detect-virt can return a nonzero status when it finds no virtualization. That alone isn’t a setup failure.
Containers can block the kernel tools needed for sandboxing. Use a virtual machine if your container host blocks the chosen controls.
Open the installed manuals. Press q to leave each page:
man systemd.service
man systemd.unit
man systemd.exec
man systemd.timer
Check each directive in these manuals before adding it. The linked upstream manuals describe current upstream behavior. They may include options your distribution doesn’t ship.
Define the lab’s needs:
| Requirement | Lab value |
|---|---|
| Executable | /usr/local/libexec/tbrt-lab-task |
| Runtime | /bin/sh and /usr/bin/date |
| Identity | Dedicated tbrt-lab account and group |
| Persistent writes | /var/lib/tbrt-lab/runs.log |
| Configuration | None |
| Credentials | None |
| Network access | None |
| Successful completion | Exit status 0 |
| Execution limit | One minute |
For a real workload, add its config paths, credential sources, network destinations, and retry behavior. A failed task may have done some work before it exits. Automatic retries need to be safe.
Build Hardened systemd Service Units From a Working Baseline
2. Create an isolated account and timestamp workload
Check the required commands and executable paths:
command -v useradd
command -v nologin
command -v install
command -v sudo
command -v tee
command -v getent
command -v false
test -x /bin/sh
test -x /usr/bin/date
test -x /usr/sbin/nologin
test -x /usr/bin/false
The test commands are silent on success. If a path doesn’t exist, resolve it first. Use its verified absolute path throughout.
Check for name and path conflicts:
getent passwd tbrt-lab
getent group tbrt-lab
systemctl show tbrt-lab.service -p LoadState -p FragmentPath
systemctl show tbrt-lab.timer -p LoadState -p FragmentPath
sudo ls -ld /etc/systemd/system/tbrt-lab.service.d
sudo ls -ld /etc/systemd/system/tbrt-lab.timer.d
sudo ls -ld /var/lib/tbrt-lab
sudo ls -l /usr/local/libexec/tbrt-lab-task
For a fresh lab, the account lookups should return no entry. The units should report LoadState=not-found, and the lab-specific paths should be absent.
Stop if anything already exists. Choose a different prefix and replace every lab name consistently. Don’t reuse an existing account or unit.
Create the account and directories:
sudo useradd --system --user-group --no-create-home \
--home-dir /nonexistent --shell /usr/sbin/nologin tbrt-lab
sudo install -d -o tbrt-lab -g tbrt-lab -m 0750 /var/lib/tbrt-lab
sudo install -d -o root -g root -m 0755 /usr/local/libexec
--system creates a system account. --no-create-home skips home directory creation, and nologin blocks normal interactive login.
Create the workload:
# /usr/local/libexec/tbrt-lab-task
sudo tee /usr/local/libexec/tbrt-lab-task >/dev/null <<'EOF'
#!/bin/sh
set -eu
/usr/bin/date -u '+%Y-%m-%dT%H:%M:%SZ' >> /var/lib/tbrt-lab/runs.log
printf '%s\n' 'tbrt-lab completed'
EOF
sudo chown root:root /usr/local/libexec/tbrt-lab-task
sudo chmod 0755 /usr/local/libexec/tbrt-lab-task
The quoted heredoc delimiter keeps the script text literal. Root owns the script so the service account can’t change its own executable.
Test it under the dedicated account:
sudo -u tbrt-lab /usr/local/libexec/tbrt-lab-task
sudo tail -n 1 /var/lib/tbrt-lab/runs.log
Expected completion message:
tbrt-lab completed
The second command should show one UTC timestamp. Its value depends on when you run the command.
3. Write, validate, and run the minimal service
Create the base unit:
# /etc/systemd/system/tbrt-lab.service
sudo tee /etc/systemd/system/tbrt-lab.service >/dev/null <<'EOF'
[Unit]
Description=Harmless TBRT hardening lab task
[Service]
Type=oneshot
User=tbrt-lab
Group=tbrt-lab
ExecStart=/usr/local/libexec/tbrt-lab-task
TimeoutStartSec=1min
EOF
Type=oneshot suits a finite task. TimeoutStartSec=1min caps its runtime. Oneshot services don’t have a start timeout by default.
Use an explicit executable path. ExecStart= doesn’t process shell redirects or pipelines unless you invoke a shell. See systemd.service.
Validate before loading:
sudo systemd-analyze verify /etc/systemd/system/tbrt-lab.service
echo $?
A successful check should return status 0, usually without output. Read any messages, including warnings that don’t prevent loading.
Then run the service:
sudo systemctl daemon-reload
sudo systemctl start tbrt-lab.service
systemctl show tbrt-lab.service \
-p Type -p ActiveState -p Result -p ExecMainStatus
sudo tail -n 2 /var/lib/tbrt-lab/runs.log
sudo journalctl -u tbrt-lab.service -b -n 20 --no-pager
Expected property values, with ordering determined by your host:
Type=oneshot
ActiveState=inactive
Result=success
ExecMainStatus=0
The log should contain another timestamp, and the journal should contain tbrt-lab completed.
An inactive state is normal after this task finishes. Leave RemainAfterExit disabled so future timer events can run it again.
4. Add bounded retries without creating a success loop
Create a named restart drop-in:
sudo install -d -o root -g root -m 0755 \
/etc/systemd/system/tbrt-lab.service.d
# /etc/systemd/system/tbrt-lab.service.d/10-restart.conf
sudo tee /etc/systemd/system/tbrt-lab.service.d/10-restart.conf >/dev/null <<'EOF'
[Unit]
StartLimitIntervalSec=10min
StartLimitBurst=3
[Service]
Restart=on-failure
RestartSec=30s
EOF
Start limits belong in [Unit]. Restart policy and delay belong in [Service].
| Workload | Restart choice | Trade-off |
|---|---|---|
| Scheduled task that should wait until its next schedule | Restart=no | A failed run gets no immediate automatic retry |
| Retry-safe scheduled task | Restart=on-failure | A partial operation may run again |
| Long-running foreground process | Often Restart=on-failure | Clean exits don’t trigger a restart |
| Long-running process that must return after clean exits too | Consider Restart=always | Intentional clean exits also restart it |
Restart=always and Restart=on-success are invalid for Type=oneshot. An explicit systemctl stop doesn’t cause an automatic restart. See restart policy behavior.
Validate and test:
sudo systemd-analyze verify /etc/systemd/system/tbrt-lab.service
sudo systemctl daemon-reload
# Clear lab counters before this deliberate test.
sudo systemctl reset-failed tbrt-lab.service
sudo systemctl start tbrt-lab.service
systemctl show tbrt-lab.service \
-p Restart -p RestartUSec -p StartLimitIntervalUSec -p StartLimitBurst
sudo tail -n 1 /var/lib/tbrt-lab/runs.log
Expect Restart=on-failure, a 30-second delay, and a burst limit of three starts per ten minutes.
Start limits also count manual starts. Later lab commands reset the counters before each planned test. This keeps earlier successful runs from skewing the result.
Once the service hits the limit, further starts are blocked until the interval passes. The interval expiring does not queue a pending retry; after it passes, a new manual, timer, or socket activation may start the unit again, and restart logic is available again.
Fix the cause, then reset the failed state and request a start. See systemd.unit start limits.
Add Filesystem and Privilege Restrictions Separately
5. Restrict filesystem access and preserve one state directory
Add the filesystem drop-in:
# /etc/systemd/system/tbrt-lab.service.d/20-filesystem.conf
sudo tee /etc/systemd/system/tbrt-lab.service.d/20-filesystem.conf >/dev/null <<'EOF'
[Service]
ProtectSystem=strict
ProtectHome=yes
PrivateTmp=yes
ReadWritePaths=/var/lib/tbrt-lab
UMask=0027
EOF
Validate and repeat the original workload:
sudo systemd-analyze verify /etc/systemd/system/tbrt-lab.service
sudo systemctl daemon-reload
sudo systemctl reset-failed tbrt-lab.service
sudo systemctl start tbrt-lab.service
systemctl show tbrt-lab.service \
-p Result -p ExecMainStatus -p ProtectSystem \
-p ProtectHome -p PrivateTmp -p ReadWritePaths -p UMask
sudo tail -n 1 /var/lib/tbrt-lab/runs.log
The timestamp must still advance. If it doesn’t, stop here and inspect the journal before adding more limits.
| Setting | Effect | What can break |
|---|---|---|
ProtectSystem=strict | Makes most of the filesystem hierarchy read-only | State, cache, or log writes outside allowed paths |
ReadWritePaths=/var/lib/tbrt-lab | Allows writes there within the filesystem sandbox | Missing directories or ordinary permissions still block writes |
ProtectHome=yes | Makes /home, /root, and /run/user inaccessible | Home-based configuration, sockets, and credentials |
PrivateTmp=yes | Gives the service private temporary-directory views | Temporary-file exchange with other services |
UMask=0027 | Restricts permissions on newly created files | Consumers expecting wider access |
ReadWritePaths= doesn’t create directories or grant ownership. It also can’t make an underlying read-only filesystem writable.
Only /var/lib/tbrt-lab needs persistent write access here. Private temporary directories remain available through PrivateTmp=yes. Don’t allow all of /var just to solve one failed write.
These settings don’t prevent reading every accessible file or block network access. Those need separate checks. See filesystem namespace settings.
6. Prevent privilege gains and restrict kernel interfaces
Add the next set of controls:
# /etc/systemd/system/tbrt-lab.service.d/30-privileges.conf
sudo tee /etc/systemd/system/tbrt-lab.service.d/30-privileges.conf >/dev/null <<'EOF'
[Service]
NoNewPrivileges=yes
ProtectKernelTunables=yes
ProtectKernelModules=yes
ProtectControlGroups=yes
RestrictSUIDSGID=yes
EOF
Retest:
sudo systemd-analyze verify /etc/systemd/system/tbrt-lab.service
sudo systemctl daemon-reload
sudo systemctl reset-failed tbrt-lab.service
sudo systemctl start tbrt-lab.service
systemctl show tbrt-lab.service \
-p Result -p ExecMainStatus -p NoNewPrivileges \
-p ProtectKernelTunables -p ProtectKernelModules \
-p ProtectControlGroups -p RestrictSUIDSGID
sudo tail -n 1 /var/lib/tbrt-lab/runs.log
Expect another successful timestamp write and enabled properties.
NoNewPrivileges=yes prevents the process and its child processes from gaining privileges through execution. Programs that need setuid helpers or file capabilities can fail.
The kernel controls restrict access to kernel settings, module operations, and control-group management. That suits a timestamp task. A host-management agent needs a different review.
For a real service:
- Move required config out of hidden home directories into a suitable service config path.
- Grant only the read access required for config.
- Use a credential mechanism supported by the installed systemd version.
- Keep secrets out of unit text, command arguments, logs, and screenshots.
I’d skip stronger memory and namespace limits unless the workload supports them. MemoryDenyWriteExecute= can break JIT runtimes. RestrictNamespaces= can break programs that create their own sandboxes.
Network limits need another test stage. PrivateNetwork= or address-family limits can block required connections. The local lab needs no network dependency, but these filesystem controls don’t enforce a network ban.
Schedule the Hardened Task and Check Timer Behavior
7. Validate the calendar before enabling the timer
Check the host clock and how systemd reads the calendar expression:
timedatectl status
systemd-analyze calendar daily
daily uses the host’s local timezone unless the expression specifies one. Confirm that timezone before choosing your schedule.
To inspect an explicit UTC alternative:
systemd-analyze calendar '*-*-* 00:00:00 UTC'
Keep the timer and the checked expression consistent. Calendar syntax is documented in systemd.time.
Create the daily timer:
# /etc/systemd/system/tbrt-lab.timer
sudo tee /etc/systemd/system/tbrt-lab.timer >/dev/null <<'EOF'
[Unit]
Description=Daily TBRT hardening lab task
[Timer]
OnCalendar=daily
Persistent=true
RandomizedDelaySec=15min
Unit=tbrt-lab.service
[Install]
WantedBy=timers.target
EOF
Validate both files and immediately check the result:
sudo systemd-analyze verify \
/etc/systemd/system/tbrt-lab.service \
/etc/systemd/system/tbrt-lab.timer
echo $?
Expect status 0 after resolving relevant errors and warnings.
Enable and start the timer:
sudo systemctl daemon-reload
sudo systemctl enable --now tbrt-lab.timer
systemctl is-enabled tbrt-lab.timer
systemctl is-active tbrt-lab.timer
systemctl list-timers --all tbrt-lab.timer
systemctl status tbrt-lab.timer --no-pager
Expected results:
enabled
active
The timer listing should identify tbrt-lab.service and show its next activation.
Three details affect scheduling:
RandomizedDelaySec=15minadds a random delay to spread scheduled work.AccuracySec=lets systemd group timer events. Its documented default is one minute.Persistent=truecan cause a catch-up activation when a calendar event was missed while the timer was inactive.
Persistent catch-up doesn’t replay every missed occurrence. It also doesn’t track successful completion. A reconciliation task must find its own outstanding work.
If the service is already active when the timer fires, systemd doesn’t launch another instance of that same service. This includes a oneshot left active with RemainAfterExit=yes. See systemd.timer.
Inspect Effective Settings and Prove the Sandbox Works
8. Inspect the loaded files, properties, and security assessment
Read the combined config:
systemctl cat tbrt-lab.service
systemctl cat tbrt-lab.timer
systemctl show tbrt-lab.service \
-p FragmentPath -p DropInPaths -p User -p Group \
-p Restart -p RestartUSec \
-p StartLimitIntervalUSec -p StartLimitBurst \
-p ProtectSystem -p ProtectHome -p ReadWritePaths \
-p NoNewPrivileges -p Result -p ExecMainStatus
sudo journalctl -u tbrt-lab.service -b -n 40 --no-pager
sudo systemd-analyze security tbrt-lab.service
systemctl cat identifies the base file and named drop-ins. After daemon-reload, use systemctl show to inspect selected loaded properties. See systemctl.
Expect these service drop-ins:
10-restart.conf
20-filesystem.conf
30-privileges.conf
The security assessment flags exposure based on the settings. It doesn’t prove the program is secure or that each limit works at runtime. Use it to find controls worth reviewing, then test them. See systemd-analyze.
9. Compare a permitted host write with a denied sandbox write
Pause scheduling before causing a deliberate failure:
sudo systemctl disable --now tbrt-lab.timer
sudo systemctl stop tbrt-lab.service
sudo ls -ld /var/lib/tbrt-lab-deny-test
The disposable probe directory must be absent. If it exists, stop and choose another unique path throughout this test.
Create it with permissions that allow the service account to write:
sudo install -d -o tbrt-lab -g tbrt-lab -m 0750 \
/var/lib/tbrt-lab-deny-test
sudo -u tbrt-lab /bin/sh -c \
'printf "%s\n" outside-ok > /var/lib/tbrt-lab-deny-test/outside.txt'
sudo cat /var/lib/tbrt-lab-deny-test/outside.txt
Expected baseline:
outside-ok
This shows that normal ownership and directory permissions permit the write.
Now add a temporary service probe:
# /etc/systemd/system/tbrt-lab.service.d/90-deny-test.conf
sudo tee /etc/systemd/system/tbrt-lab.service.d/90-deny-test.conf >/dev/null <<'EOF'
[Service]
Restart=no
ExecStartPre=/bin/sh -c 'echo sandbox-probe > /var/lib/tbrt-lab-deny-test/inside.txt'
EOF
sudo systemd-analyze verify /etc/systemd/system/tbrt-lab.service
sudo systemctl daemon-reload
sudo systemctl reset-failed tbrt-lab.service
sudo systemctl start tbrt-lab.service
The start is expected to fail. Restart=no prevents repeated probe attempts.
Inspect the actual result:
systemctl show tbrt-lab.service -p Result -p ExecMainStatus
sudo journalctl -u tbrt-lab.service -b -n 30 --no-pager
sudo ls -la /var/lib/tbrt-lab-deny-test
Expect the shell’s attempted write to fail because the target is read-only inside the service. The inside.txt marker should be absent.
A generic permission error alone doesn’t prove ProtectSystem enforcement. Check the actual journal message. A namespace setup failure means the probe never reached the intended write.
If the outside write succeeds and the inside write fails on a read-only target, unchanged ownership helps isolate the cause. That points to the filesystem sandbox. This test proves the limit for this operation. It doesn’t test every directive.
Warning: The following commands delete only the named disposable probe files. Inspect their contents first.
sudo systemctl stop tbrt-lab.service
sudo rm /etc/systemd/system/tbrt-lab.service.d/90-deny-test.conf
sudo rm /var/lib/tbrt-lab-deny-test/outside.txt
sudo rmdir /var/lib/tbrt-lab-deny-test
sudo systemctl daemon-reload
sudo systemd-analyze verify /etc/systemd/system/tbrt-lab.service
sudo systemctl reset-failed tbrt-lab.service
sudo systemctl start tbrt-lab.service
sudo tail -n 1 /var/lib/tbrt-lab/runs.log
The original timestamp task must succeed again. If an unexpected file prevents rmdir, inspect it instead of switching to recursive deletion.
10. Verify failed-run retries and start-rate limiting
Keep the timer disabled. Replace only the executable through another temporary drop-in:
# /etc/systemd/system/tbrt-lab.service.d/90-failure-test.conf
sudo tee /etc/systemd/system/tbrt-lab.service.d/90-failure-test.conf >/dev/null <<'EOF'
[Service]
ExecStart=
ExecStart=/usr/bin/false
EOF
sudo systemd-analyze verify /etc/systemd/system/tbrt-lab.service
sudo systemctl daemon-reload
sudo systemctl reset-failed tbrt-lab.service
sudo systemctl start tbrt-lab.service
The empty ExecStart= clears the original command before setting its replacement. The initial start is expected to fail.
Wait about 100 seconds, then inspect:
sudo journalctl -u tbrt-lab.service -b --since '-3 minutes' --no-pager
systemctl show tbrt-lab.service \
-p ActiveState -p SubState -p Result -p NRestarts \
-p Restart -p RestartUSec -p StartLimitBurst
With clean counters, three starts are allowed within ten minutes. Retries are separated by 30 seconds. The next attempted start should hit the limit.
Use the real journal timestamps to confirm the behavior. Don’t infer the number of executed attempts from one restart-counter property alone.
Restore the workload:
Warning: This removes the named failure-test override and stops any pending automatic retry.
sudo systemctl stop tbrt-lab.service
sudo rm /etc/systemd/system/tbrt-lab.service.d/90-failure-test.conf
sudo systemctl daemon-reload
sudo systemd-analyze verify /etc/systemd/system/tbrt-lab.service
sudo systemctl reset-failed tbrt-lab.service
sudo systemctl start tbrt-lab.service
systemctl show tbrt-lab.service -p ActiveState -p Result -p ExecMainStatus
sudo tail -n 1 /var/lib/tbrt-lab/runs.log
Expect one successful completion followed by ActiveState=inactive. Successful oneshots shouldn’t enter a restart loop.
11. Observe persistent catch-up on a short lab schedule
Use a temporary timer override:
sudo install -d -o root -g root -m 0755 \
/etc/systemd/system/tbrt-lab.timer.d
# /etc/systemd/system/tbrt-lab.timer.d/90-catchup-test.conf
sudo tee /etc/systemd/system/tbrt-lab.timer.d/90-catchup-test.conf >/dev/null <<'EOF'
[Timer]
OnCalendar=
OnCalendar=*-*-* *:*:00
RandomizedDelaySec=0
AccuracySec=1s
EOF
systemd-analyze calendar '*-*-* *:*:00'
sudo systemd-analyze verify /etc/systemd/system/tbrt-lab.timer
sudo systemctl daemon-reload
sudo systemctl reset-failed tbrt-lab.service
sudo systemctl start tbrt-lab.timer
systemctl list-timers --all tbrt-lab.timer
The empty OnCalendar= clears the daily expression. The replacement schedules activation at each minute boundary.
Wait until one scheduled run completes, then inspect and stop the timer:
sudo journalctl -u tbrt-lab.service --since '-2 minutes' --no-pager
sudo tail -n 3 /var/lib/tbrt-lab/runs.log
sudo systemctl stop tbrt-lab.timer
date -u
Leave it stopped across at least two minute boundaries. Then restart it:
sudo systemctl reset-failed tbrt-lab.service
sudo systemctl start tbrt-lab.timer
systemctl list-timers --all tbrt-lab.timer
sudo journalctl -u tbrt-lab.service --since '-3 minutes' --no-pager
sudo tail -n 5 /var/lib/tbrt-lab/runs.log
Look for one catch-up activation after restarting. Check promptly because normal minute-based runs will resume too.
Restore the daily schedule:
Warning: This deletes the temporary timer override and restarts the timer with its daily schedule.
sudo systemctl stop tbrt-lab.timer
sudo systemctl stop tbrt-lab.service
sudo rm /etc/systemd/system/tbrt-lab.timer.d/90-catchup-test.conf
sudo rmdir /etc/systemd/system/tbrt-lab.timer.d
sudo systemctl daemon-reload
sudo systemd-analyze verify \
/etc/systemd/system/tbrt-lab.service \
/etc/systemd/system/tbrt-lab.timer
sudo systemctl reset-failed tbrt-lab.service
sudo systemctl enable --now tbrt-lab.timer
systemctl cat tbrt-lab.timer
systemctl list-timers --all tbrt-lab.timer
Confirm that OnCalendar=daily and the 15-minute randomized delay are restored.
Fix the Failing Layer and Roll Back One Change
12. Match common failures to a specific check
| Symptom | Likely layer | Targeted action |
|---|---|---|
| Unknown directive or ignored setting | Syntax or installed-version support | Read the local manual, correct its section, verify, and reload |
| State write fails | Ownership or filesystem sandbox | Inspect the directory and effective writable-path allowance |
| Configuration disappears | Home-directory isolation or access permissions | Identify its exact path and provide a suitable service configuration path |
| Retries stop | Start-rate limit | Fix the cause, reset counters, and start deliberately |
| Timer is enabled but inactive | Activation state | Start it with enable --now |
| Later timer activations do nothing | Service remains active | Inspect state and remove unintended RemainAfterExit=yes |
For state-write failures, inspect both layers:
sudo ls -ld /var/lib/tbrt-lab
sudo ls -l /var/lib/tbrt-lab/runs.log
systemctl show tbrt-lab.service -p ProtectSystem -p ReadWritePaths
sudo journalctl -u tbrt-lab.service -b -n 30 --no-pager
The directory should belong to tbrt-lab:tbrt-lab. Correct only this lab directory if its ownership is wrong:
sudo chown tbrt-lab:tbrt-lab /var/lib/tbrt-lab
sudo chmod 0750 /var/lib/tbrt-lab
If the existing log has incorrect ownership, inspect it and correct that file separately. Avoid recursive permission changes.
For an invalid oneshot restart policy, edit the known restart drop-in:
sudoedit /etc/systemd/system/tbrt-lab.service.d/10-restart.conf
Use Restart=no or Restart=on-failure, then verify and reload. Don’t move start-limit settings into [Service].
For an inactive timer:
sudo systemctl enable --now tbrt-lab.timer
systemctl status tbrt-lab.timer --no-pager
For unexpected service activity:
systemctl show tbrt-lab.service \
-p ActiveState -p SubState -p RemainAfterExit
systemctl cat tbrt-lab.service
Inspect a running task before requesting another activation. A completed recurring oneshot shouldn’t be held active by RemainAfterExit=yes.
13. Remove the last hardening increment safely
If 30-privileges.conf caused the failure, roll back that file alone.
Warning: This stops the lab schedule and service, then removes the named privilege-hardening drop-in.
sudo systemctl disable --now tbrt-lab.timer
sudo systemctl stop tbrt-lab.service
sudo cat /etc/systemd/system/tbrt-lab.service.d/30-privileges.conf
sudo rm /etc/systemd/system/tbrt-lab.service.d/30-privileges.conf
sudo systemctl daemon-reload
sudo systemd-analyze verify /etc/systemd/system/tbrt-lab.service
sudo systemctl reset-failed tbrt-lab.service
sudo systemctl start tbrt-lab.service
systemctl show tbrt-lab.service -p Result -p ExecMainStatus
sudo tail -n 1 /var/lib/tbrt-lab/runs.log
Expect success under the previous filesystem-hardening stage. Only then restore scheduling:
sudo systemctl enable --now tbrt-lab.timer
This rollback leaves the privilege controls removed. To restore them, recreate the exact file from step 6 and repeat its checks.
For complete lab removal, first stop both units. Remove only the named unit files, remaining known drop-ins, and workload script. Reload systemd afterward. Save any state you want before deleting the dedicated directory, account, and group.
Avoid broad wildcard deletion. A named drop-in keeps the rollback small enough to review.
14. Record acceptance checks and verify reboot persistence
Before adapting this design for production, record:
- Distribution, systemd package version, kernel, and supported directives.
- Executable ownership, service identity, config paths, and credential access.
- Required writable directories and successful writes from within the service.
- Actual denied-operation results and failure reasons.
- Retry safety, delay, and observed start-limit behavior.
- Timezone, scheduling tolerance, catch-up expectations, and overlap behavior.
- Effective files, relevant logs, and exact rollback paths.
Keep this lab separate from existing production units. Apply reviewed changes during a suitable maintenance window.
Before rebooting, confirm the timer is enabled:
systemctl is-enabled tbrt-lab.timer
systemctl list-timers --all tbrt-lab.timer
Warning: The next command reboots the lab host and disconnects SSH. Save your work first.
sudo systemctl reboot
Reconnect, then check:
systemctl is-enabled tbrt-lab.timer
systemctl is-active tbrt-lab.timer
systemctl list-timers --all tbrt-lab.timer
sudo systemctl reset-failed tbrt-lab.service
sudo systemctl start tbrt-lab.service
systemctl show tbrt-lab.service -p Result -p ExecMainStatus
sudo tail -n 1 /var/lib/tbrt-lab/runs.log
The timer should be enabled and active after boot. The manual start separately proves that the hardened workload can still write its state.
Wrapping Up
A dedicated account, explicit write allowance, and tested restart policy make faults easier to trace. I’d keep each hardening group in its own drop-in. A smaller rollback is worth the extra files.
| Step | Action | Applies To |
|---|---|---|
| Establish a baseline | Verify identity and successful execution | Every custom service |
| Add restrictions | Retest required writes after each increment | Hardened services |
| Prove behavior | Test denial, retries, and catch-up | This lab and reviewed adaptations |
| Check persistence | Confirm timer activity and a successful write after reboot | Scheduled deployment |
After reboot, confirm the timer is active and the hardened service appends a fresh timestamp. That’s the final acceptance check.