• 1 Post
  • 182 Comments
Joined 3 years ago
cake
Cake day: August 11th, 2023

help-circle





  • First find a way to track your costs. Measure how much it costs to give it away (to your self and people that support you starting it as a buisness). Do market analysis based on those number. How much do you need to support both your operatiinal costs of hosting and devoloper time paid with a fair wage, and what does compare to how much consumers are currently paying. If that looks good invest in an accountant and lawyer to see what the buisness plan viability might look like and set up a LLC. From there look for partners that can help you create a sustainable buisness and fill in skill gaps you dont have, or hire out the rest.

    Thats it.

    Even better make it a community coop so that the skilled labor and community can have actual ownership in the org to help balance out thr risk and encourage litteral buyin.


  • I’m 100% biased towards this. It’s really cool to see a system get frankenstiened or frankenstein a system from code. It’s like getting to leap frog from all of their work without needing to ever explicitly knowing or working with each other.

    The real deciding factor isn’t whether to automate or not but what is the expected technical skills of who expect to contribute or maintain it the future.

    So I personally layer levels to it prefering the common automation where applicable. So tofu for infra and spin up. Ansible for OS config and network equipment. Kubernetes for everything else with a fallback to tofu for some apps that have no k8s API but a good tofu provider.

    With that people can actually piecemeal what they want and where they want it if some underlying assumption changed (it’s no longer bare metal or aws, it’s Ubuntu or SuSE now, it’s RKE2 or it’s GKE now etc)

    That said for bare metal I actually skip tofu and do a cloud native bare metal management service like TinkerBell or Metal3, and that’s really only if it’s worth it.

    The other, when you see it work it’s beautiful, thing about the approach. GitOps. Watching what was a thousand shadow IT ops become PRs through automated testing and review is really magical. Coming from a IT back ground for life saving infrastructure the stress difference between the old hat way and this is really hard to understate.


  • This is truly an amazing guide on building resilient systems. It’s so common for people to ignore human based failure domains too!

    That said some of the early recommendations suggest fully mapping dependencies such as system accounts, payment accounts, etc. I’ve looked at that from the “asCode” side but always felt quite lacking. With digital twin modeling of an orginization being a highly academic work (how do you truly capture the reality right?) and smart contracts such DAOs being a heavy tool for enforcement. I was just wondering if you have recommendation for anything or templates?

    If not I might try to templatize something based on those lessons myself just to see if there is an quick start way for people to do this.











  • I would recommend 4-5 nodes. 5 if you want true high availability. 4 still requires some intervention in case of failure.

    Just because it’s bare metal. Got to think of your Mean Time to Repair (MTTR) which is to say if a whole node goes bust how long will it take to potentially order and install a new one.

    If you go kubernetes (k8s) I would recommend rke2 or k3s. They are really straightforward setups and pretty enterprise ready out of the box.

    If you have a hard requirement for Ceph I would recommend doing Rook-Ceph which makes deploying and management a lot easier by letting k8s handle it. For simpler but less performant (in my testing) persistent volumes (PVs) like ceph Longhorn is really easy to deploy and manage.

    For backups Velero is really nice for apps in your cluster, since it can be done per namespace and include PV data too. Rke2/k3s both have nice etcd (the backend data base for k8s) snapshoting and backup tools too for full disaster recovery.

    Rke2/k3s both have ways to auto deploying charts from the filesystem too https://docs.rke2.io/add-ons/helm

    This is a good stepping stone for GitOps imho. If that matters to you at all. Starting with just having a git dir for these files, then later doing some like ArgoCD

    I would also recommend, since you are looking at hyper converged storage have dedicated network lines for it is generally recommended. So create a bond of two ports per node just for storage, tag them with their own vlan, and in your setup of rook or longhorn specific that vlan interface as the device for data to flow.

    Pxe boot is also nice at this scale, either setup on your router (OpenWrt has decent support), you maintance laptop/machine, and/or do something like Tinkerbell (cloud native pxe from your k8s cluster!). It’s just nice to be able to blow away a node and rebuild if you are tinkering a lot.

    Remember cattle not pets, and welcome to the range cowpoke!