I usually connect with my server via ssh in a terminal and run basic commands. What’s a better, more efficient and modern way of doing that? Especially considering ai and documentation along the way? I wonder if there’s a better approach than “connect from remote and act local”. Is there a method to “code local and push to remote”?
I use a fedora server with podman, caddyfile and vi.
If you’re connecting from a system that’s a long distance away from your server in terms of latency, mosh can be more comfortable to use than ssh.
Mosh is pretty good!
I wait for something to break. Then I yell “God fucking dammit, I don’t have time for this right now!” and spend an hour triaging things before I just try docker down, pull, up and everything works.
same. or i just df and see / 100%
Early on I wrote a script for my user account to run df and save the output to a file, then a second script to read that file and if the drive was full send me an email.
First, it failed because it couldn’t send the email because the email service failed under full disk conditions.
Second, it failed because it couldn’t write the file to disk because the disk was full.
Third, it failed because outbound SMTP was blocked by my ISP.
I learned a lot about how well you can fail if you really put your mind to it. Now I have Home Assistant grabbing the disk stats for my machines and flagging anything over 90%.
This is oddly inspirational
Poorly.
Hear hear
Hell yeah
I just use ssh and manually type commands. Keep doing it and you will get good and it will become second nature.
No need to burn tokens on basic tasks.
Master your tools. Learn awk, sed, just, etc.
Some shell customisation can help. For example I’m fond of zsh-auto-suggestions and skim. Makes me quicker.
Ansible is one way to “code local, push to remote(s)”. Can define so-called playbooks, which is basically just a script, to do reoccuring tasks like e.g. updates.
Ansible is great, I just wish it would automatically cleanup stuff on the server when I remove parts of the config. I know it’s not the goal, but I dream about a tool that works this way. I get many leftover stuff over years
That sounds like NixOS to me.
I have yet to try it but apparently
We fixed that with roles that manage both install and uninstall
That works but that’s only useful if you have many machines. I have a single server so it’s the same burden as doing it manually. If only the uninstall step could be completely automated by just reverting the install step
Not quite, if you write the role to manage the life cycle of a bit of software then uninstall is part of that life cycle.
But I do see that investment in time looks more then just do it by hand
To be fair to Ansible, that’s probably a packaging issue.
If the 1st run of an application creates a bunch of files and folders, then the packager won’t know about them and Ansible’s just relying on that.
But yeah, for 1 machine I’d find Ansible overkill (not a problem per se), but really helps when you have 5 Rasperry Pi Zeros scattered around the house 😉
I generally find writing a good role takes longer than doing it manually. Especially as you have to keep testing failure conditions as you think of them.
However, the point is, that once you’re finished with the role, you never have to do that process again.
Not if you want to rebuild Not if you want to migrate to a new os Not if you build a second server Not if you move to a newcloud hosting provider
Porting to a new OS or after an update that changes how the software works is about as fast as doing it manually
Maybe it’s only one server today, but how many times would you need to rebuild it over the running lifetime?
The killer feature is that having something like ansible lets you kill your darlings, and keep servers aa cattle, not pets.
Manual setup chops isn’t a different skill here. I view ansible as a combination setup and documentation. If you’re going to figure out how, why not write down how you did it? If you’re going to write it down complete with code lines to run, why not script it?
I use ssh with tmux
This is the way
<insert Mandalorian gif here>
“code local and push remote” is only more efficient if you have a faulty connection with ssh or want to use a GUI editor that doesn’t support remote connections. What you’re doing is good. You can scoure for more “modern” ways that might have more glimmer and sparkly lights, but it won’t improve efficiency.
Declarative configuration (like docker compose) is basically “code local, push remote”, and it has a lot of benefits, including improving efficiency because you have a clearer idea of exactly what is set up on the server, to make it easier to debug or make changes.
Contrast this with just sshing in and running a bunch of commands to install something. Then coming back months later and wondering “what files did I touch? what packagea did I install? Hmm dpkg log says I added X but then later I removed it, so was it important?” Etc.
I write my NixOS configs in my PC, commit to the repo, then push it to the server. Then SSH in, and apply it. Or more likely, MOSH in.
I created git repos on my main workstation for each homelab server/service I maintain that keeps:
- documentation
- notes
- lessons learned
- scripts, configs
- runbooks
- backup details
- security audit details
- log items that need attention
- infrastructure
I just point a local LLM (offline model that runs on my workststion) into those repos and ask it to perform certain things on those servers. It can do things like update packages, install packages, make config changes, set/check permissions, read logs (and fix errors in real time), and check the health of the overall system.
I have it run pre backups before making changes, then post backups once its done.
Once changes are in place and everything is running okay, I ask it to update documentation in the repo and tag the release.
I use
opencodethat connects to allama.cppservice.opencodelets me gate the AI so that any elevated commands that it needs to run (e.g. sudo or ssh), I have to approve it. It cant just go around making changes without permission.What models are you using? What hardware are you running them on? I’m curious if I can replicate your success
Hardware I’m running:
- 8/16 AMD CPU
- 32GB system RAM
- 12GB GPU VRAM (AMD)
I’m mainly using these MoE models:
Qwen-3.6-35B-A3B
- Q4_K_M quant quality
- 128k context (conversation length before it compacts)
- Gives me about 260 prefill and 19 token gen speeds
Gemma4-26B-A4B
- Q8 quant quality
- 128k context length
- Gives me about 190 prefill and 14 token gen speeds
It’s good to get comfortable in the command line, including over ssh.
Especially considering ai and documentation along the way?
If I am going to ask AI or Google about something, I just do that to help me find the answer and then apply the answer myself. That way I learn, and can double check the AI isn’t hallucinating, at least on in obvious ways.
As for documentation, I keep a notes folder with detailed notes on manual configuration I’ve done, how and why, and the things I’ve learned along the way. I’ve found it’s both useful to help remember the things I’ve learned, and it is useful to go back to refer to.
A mix of prayer and bash scripts.
Remove bash scripts and add poorly configured logging and that’s me!
At least you have logging!
I’ve been using a program someone on Lemmy has been writing: SSHPilot. It helps keep my server list organized and can easily ssh to the machines or open up a file browser for easy drag and drop of files if I have some quick files to move around. It copies ssh keys easily as well.
I ssh into it twice a year to perform updates but otherwise i leave it alone. Server’s been chugging along for like a decade now.
Similar, but you might want to update more frequently with the huge number of critical security bugs being discovered by AI recently
I have unattended upgrades configured
Hmm I need to figure out how to do that
If you are using aptitude (apt) it is quite simple: https://ubuntu.com/server/docs/how-to/software/automatic-updates/
If you just want security updates and are fine with the default configs:
# apt install unattended-upgrades
Twice a year is crazy. I suppose most of my maintainence is adding drives (I’m a creator and I definitely totally need to archive all of my footage), but I also run a fair few apps that have needed manual update steps, so I do all my updates manually so I can fix stuff if it breaks.
nixos, mostly. It has a very steep learning curve but if it breaks I’m just “whatever, rollback now, fix properly later”.
Ansible.
That, and there is no change that isn’t done on ansible, or, when it required trying and research gets backported into a bit of ansible.
